WO2022143883A1 - Photographing method and system, and electronic device - Google Patents

Abstract

The present application provides a photographing method and system, and an electronic device, relating to the field of photographing. Pictures photographed by different devices can be flexibly and accurately switched to the electronic device for display, and the user experience is improved. The method comprises: a first device may display a first photographed picture in a display interface, wherein the first photographed picture is a picture acquired by means of a camera of the first device; after the first device establishes a network connection with N electronic devices, the first device may instruct the N electronic devices to start acquiring the photographed pictures, wherein N is an integer greater than 1; if the first device obtains a second photographed picture acquired by a second device, the first device may identify whether the second photographed image contains a preset utterance identifier which indicates that a user is speaking; if the second photographed picture contains the preset utterance identifier which indicates that the user using the second device is speaking, the first device may display the second photographed picture in the display interface.

Description

A shooting method, system and electronic device

This application claims the priority of a Chinese patent application with an application number of 202011630348.0 and an application title of "A Shooting Method, System and Electronic Device" submitted to the State Intellectual Property Office on December 30, 2020, the entire contents of which are incorporated by reference in in this application.

technical field

The present application relates to the field of photography, and in particular, to a photography method, system and electronic device.

Background technique

Currently, many users have multiple electronic devices with photographing functions set up in their home or workplace. In some scenarios, the user may need to watch shooting images shot by different electronic devices.

For example, in a meeting scene or a scene where multiple people are on a video call, each user can use one electronic device (eg, a mobile phone) to shoot. When user A uses the mobile phone 1 to speak during the conference, other users may wish to receive audio data and captured images from the mobile phone 1 by using their own mobile phones. Subsequently, when user B uses the mobile phone 2 to speak during the conference, other users may wish to use their mobile phones to receive audio data and captured images from the mobile phone 2 . In such a scenario, how to flexibly and accurately switch the shooting images of different devices to be displayed in an electronic device has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

The present application provides a shooting method, system and electronic device, which can flexibly and accurately switch the shooting images of different devices to display in the electronic device, thereby improving the user experience.

To achieve the above object, the application adopts the following technical solutions:

In a first aspect, the present application provides a shooting method, comprising: a first device can display a first shooting picture in a display interface, and the first shooting picture is a shooting picture collected by the first device using its own camera; when the first device After establishing a network connection with N electronic devices, the first device can instruct the N electronic devices to start capturing and shooting images, where N is an integer greater than 1; if the first device obtains the second device (the second device is the above N electronic devices) One of the devices) collected the second shooting picture, the first device can identify whether the second shooting picture contains a preset sounding mark, and the sounding mark is used to indicate that the user is making a sound; if the second shooting picture contains a preset sounding mark identification, indicating that the user using the second device is speaking, and the first device can display the second shooting picture in the display interface. At this time, the first device may continue to display the first photographing image (ie, the photographing image of the first device) on the display interface, or may not display the first photographing image.

It can be seen that in a distributed shooting scenario where the first device is connected to multiple electronic devices, when a user inputs an audio signal to an electronic device (for example, a second device), the second device can The way of adding a preset sounding mark in the first device enables the first device to know that a user inputs an audio signal to the second device. Furthermore, the first device may display the shooting picture carrying the sounding mark on the display interface. In this way, when different users use their respective devices to input audio signals, the first device, as the master device, can flexibly and accurately switch the captured image from the slave device with the user's voice input to the master device for display, so that the user can The shooting screen of the slave device with the user's voice input can be focused to improve the user's experience.

In a possible implementation manner, a preset button may be set in the above-mentioned display interface, and the preset button is used to synchronously display the shooting images of multiple devices; for example, the name of the preset button may be dual-view mode or multi-view mode, etc. Taking the dual-view mode as an example, before the first device acquires the second shot captured by the second device, the method further includes: in response to the user's operation of selecting a preset button, the first device may create a first window and a first window in the display interface. Two windows, wherein the first window is used to display the first shooting image collected by the first device, and the second window is used to display the shooting image collected by other devices; at this time, the first device displays the second shooting image on the display interface, Specifically, it includes: the first device displays the second shooting picture in the second window, and displays the first shooting picture in the first window. In this way, after the user selects the preset button, the first device can simultaneously display the first shooting image captured by itself and the shooting images of other devices that have the user's voice input.

In a possible implementation manner, in response to the operation of the user selecting the preset button, the method further includes: establishing a network connection between the first device and the N electronic devices; Before the first device acquires the second shooting image collected by the second device, the method further includes: the first device acquiring the third shooting image collected by the third device, and the third device is an electronic device other than the second device among the N electronic devices. device; the first device displays the third shooting picture in the second window, and displays the first shooting picture in the first window. That is to say, after the user selects the preset button, the first device can first display the shooting screen of the first connected device in the second window, and then, when the shooting screen with the user's voice input is obtained, the first device can display the shooting screen in the second window. In the second window, the displayed shooting screen is switched to the shooting screen (eg, the second shooting screen) with the user's voice input.

Of course, after the user selects the preset button, the first device may first display the shooting images of one or more other devices among the N electronic devices in the second window, which is not limited in this application.

In a possible implementation manner, if the above-mentioned second shooting picture contains a preset sounding mark, the first device displays the second shooting picture on the display interface, which specifically includes: if the second shooting picture contains a preset sounding mark The first device can determine whether the time to display the third shot in the second window exceeds the preset time; if the preset time exceeds, the first device displays the second shot in the second window.

That is to say, if the time since the last time the shooting screen was switched in the second window did not exceed the preset time, it means that the first device switched the shooting screen in the second window soon, in order to avoid frequent switching of shooting in the second window. The picture brings bad user experience to the user, and the first device may continue to display the third shooting picture of the third device in the second window. If the time since the last time the shooting screen was switched in the third shooting screen exceeds the preset time, the first device may switch the shooting screen displayed in the third shooting screen to the currently received shooting screen carrying the sounding logo.

In a possible implementation manner, after the first device displays the second shooting picture in the second window, the method further includes: the first device acquiring the third shooting picture collected by the third device; If the sound-emitting identifier is included, the first device can continue to display the second shooting picture in the second window without displaying the third shooting picture. In this way, the shooting images collected by the slave device without the user's voice input will not be displayed on the display interface of the first device (ie, the master device), and the master device can flexibly and accurately capture the shooting images collected by the slave device with the user's voice input. The screen is switched to be displayed on the master device, so that the user can focus on the shooting screen of the slave device with the user's voice input.

In a possible implementation manner, after the first device acquires the third shooting picture collected by the third device, the method further includes: if the third shooting picture contains a sounding logo, the first device can display it in the second window The third shooting screen, at this time, the second screen no longer displays the second shooting screen of the second device; or, if the third shooting screen contains a sounding mark, the first device can create a third window in the display interface, and The third shooting picture is displayed in the third window, and at this time, the second window can continue to display the second shooting picture of the second device.

In a possible implementation manner, the above-mentioned second window may include a prompt message, where the prompt message is used to prompt that the user corresponding to the second window is making a sound.

In a possible implementation manner, if the second shooting picture contains a preset sounding mark, the first device displays the second shooting picture on the display interface, including: if the second shooting picture contains a preset sounding mark , the first device switches the first shooting picture to the second shooting picture in the display interface, and at this time, the first device's own shooting picture is no longer displayed in the display interface.

In a possible implementation manner, after the first device establishes a network connection with the N electronic devices, the method further includes: the first device can detect whether a user inputs an audio signal to the first device; if a user inputs an audio signal to the first device audio signal, the first device can add a sounding mark to the collected first shooting picture; further, the first device can send the first shooting picture carrying the sounding mark to the connected N electronic devices, so that the N electronic devices The electronic device displays the first shooting picture according to the sounding identification. In this way, the slave device connected to the first device can also display the shooting picture carrying the sounding mark on the display interface, so that users in the distributed shooting scene can focus on the shooting picture of the slave device with the user's voice input.

In a possible implementation manner, the first device detects whether a user inputs an audio signal to the first device, which specifically includes: when the first device uses a camera to capture the first shot, the first device can use a microphone to detect the audio signal like a slave device. When the first device detects the audio signal, M first shooting pictures corresponding to the detected audio signal can be obtained, and M is an integer greater than 1; If the audio signal matches, the first device may determine that a user inputs an audio signal to the first device. This method of detecting whether a user inputs an audio signal to the device does not require pre-training and storage of the user's voiceprint, facial features and other information, nor does it require interaction with the server, which can reduce the complexity and cost of implementation.

In a possible implementation manner, when the first device detects an audio signal, acquiring M first shooting pictures corresponding to the detected audio signal, which specifically includes: when the first device detects that the loudness of the audio signal is greater than a predetermined When the value is set, M first shooting pictures corresponding to the detected audio signals are acquired, thereby increasing the accuracy with which the first device determines the audio signal input by the user to itself.

In a possible implementation manner, after the first device detects whether a user inputs an audio signal to the first device, the method further includes: if a user inputs an audio signal to the first device, indicating that the user speaking at this time is using the first device. the user of the device, the first device may stop displaying the second shooting picture of the second device. Of course, the first device can also continue to display the second shooting picture of the second device, which is not limited in this application.

In a second aspect, the present application provides a shooting method, which includes: after the second device establishes a network connection with the first device, the second device can use a camera to capture a shooting image, and use a microphone to detect an audio signal; when the second device detects that there is a user When the audio signal is input to the second device, the second device can add a preset sounding mark to the captured shooting picture, and the sounding mark is used to indicate that the user is making a sound; further, the second device can carry the shooting picture with the sounding mark. It is sent to the first device, so that the first device can display the shooting picture carrying the sounding mark on the display interface. In this way, when different users use their respective slave devices to input audio signals, the first device, as the master device, can flexibly and accurately switch the captured image captured by the slave device with the user's voice input to the master device for display, so that the The user can focus on the shooting screen of the slave device with the user's voice input, which improves the user's experience.

In a possible implementation manner, after the second device uses the camera to capture the captured image and uses the microphone to detect the audio signal, the method further includes: when the second device detects the audio signal, acquiring an M corresponding to the detected audio signal. M is an integer greater than 1; if the mouth shape features in the M shooting images match the detected audio signal, the second device may determine that a user inputs an audio signal to the second device.

In a possible implementation manner, when the second device detects an audio signal, acquiring M shooting pictures corresponding to the detected audio signal includes: when the second device detects that the loudness of the audio signal is greater than a preset value , and acquire M shooting pictures corresponding to the detected audio signal.

In a possible implementation manner, after the second device establishes a network connection with the first device, the method further includes: the second device acquires the shooting picture of the first device; The second device displays the shooting screen of the first device on the display interface. That is to say, the second device can also display a shooting picture carrying a sounding logo, so that users in the distributed shooting scene can focus on the shooting picture of the slave device with the user's voice input.

In a possible implementation manner, the display interface of the second device may also include a first window and a second window, wherein the second device displays the shooting picture of the first device in the display interface, including: the second device may The shooting picture of the first device is displayed in the first window, and the shooting picture of the second device is displayed in the second window.

In a third aspect, the present application provides an electronic device (such as the above-mentioned first device), comprising: a display screen, a communication module, one or more processors, one or more cameras, one or more memories, and one or more A computer program; wherein the processor is coupled to the communication module, the display screen and the memory, the above-mentioned one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes one or more stored in the memory. A computer program to cause an electronic device to execute the photographing method in any of the above-mentioned aspects.

In a fourth aspect, the present application provides an electronic device (such as the above-mentioned second device), comprising: a display screen, a communication module, one or more processors, one or more cameras, one or more memories, and one or more A computer program; wherein, the processor is coupled to the communication module and the memory, the above-mentioned one or more computer programs are stored in the memory, and when the electronic device runs, the processor executes the one or more computer programs stored in the memory, So that the electronic device executes the photographing method in any one of the above-mentioned aspects.

In a fifth aspect, the present application provides a distributed shooting system, including the first device in the third aspect and the second device in the fourth aspect. Wherein, the first device can display a first photographing picture in the display interface, and the first photographing picture is a photographing picture collected by the first device; after the first device establishes a network connection with the N electronic devices, The first device instructs the N electronic devices to start capturing and shooting images, where N is an integer greater than 1; the second device may also use a microphone to detect audio signals when using a camera to capture a shooting image; When the device detects that a user inputs an audio signal to the second device, the second device adds a preset sounding mark to the captured shooting picture (for example, the second shooting picture), and the sounding mark is used to indicate that the user is sound; the second device sends the second shooting picture carrying the sounding logo to the first device; after the first device obtains the second shooting picture, if the second shooting picture contains a pre- If the sounding mark is set, the first device displays the second shooting picture in the display interface.

In a sixth aspect, the present application provides a computer-readable storage medium, comprising computer instructions, when the computer instructions are executed on the above-mentioned first device or the second device, the first device or the second device is made to perform any of the above-mentioned aspects. the shooting method described.

In a seventh aspect, the present application provides a computer program product that, when the computer program product runs on the first device or the second device, enables the first device or the second device to execute the shooting method described in any one of the above aspects.

An eighth aspect provides a graphical user interface (graphical user interface, GUI) on an electronic device, the electronic device has a display screen, a camera, a memory, and one or more processors, the one or more processors are used to execute the One or more computer programs in the memory, and the graphical user interface includes a possible design of the electronic device to execute the first aspect and any one of the first aspect or the second aspect and any one of the second aspect and the second aspect. When the technical solution is used, the graphical user interface is displayed.

It can be understood that the electronic equipment, distributed shooting system, computer-readable storage medium and computer program product provided by the above aspects are all applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to above. The beneficial effects in the corresponding method provided in this article will not be repeated here.

Description of drawings

1 is a schematic diagram of the architecture of a distributed shooting system provided by an embodiment of the present application;

FIG. 2 is a schematic diagram 1 of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 3 is a second schematic diagram of an application scenario of a shooting method provided by an embodiment of the present application;

4 is a schematic diagram 3 of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram 1 of an electronic device according to an embodiment of the present application;

6 is a schematic structural diagram of an operating system in an electronic device provided by an embodiment of the present application;

FIG. 7 is a fourth schematic diagram of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 8 is a schematic diagram 5 of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 9 is a schematic diagram 6 of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 10 is a seventh schematic diagram of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 11 is a schematic diagram eight of an application scenario of a shooting method provided by an embodiment of the present application;

12 is a schematic diagram 9 of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 13 is a schematic diagram ten of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 14 is a schematic diagram eleventh of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 15 is a schematic diagram twelve of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 16 is a schematic diagram thirteen of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 17 is a fourteenth schematic diagram of an application scenario of a shooting method provided by an embodiment of the present application;

18 is a fifteenth schematic diagram of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 19 is a sixteenth schematic diagram of an application scenario of a shooting method provided by an embodiment of the present application;

FIG. 20 is a second schematic structural diagram of an electronic device according to an embodiment of the present application;

FIG. 21 is a third schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

The implementation of this embodiment will be described in detail below with reference to the accompanying drawings.

A shooting method provided by the embodiment of the present application can be applied to the distributed shooting system 200 shown in FIG. 1 . As shown in FIG. 1 , the distributed photographing system 200 may include a master device (master) 101 and N slave devices (slaves) 102 , where N is an integer greater than 0. Communication between the master device 101 and any one of the slave devices 102 may be performed in a wired manner or wirelessly.

Exemplarily, a wired connection may be established between the master device 101 and the slave device 102 using a universal serial bus (USB). For another example, the master device 101 and the slave device 102 can use the global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (code division multiple access) multiple access, CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), Bluetooth, wireless fidelity (Wi-Fi), NFC, voice over Internet protocol (VoIP), and communication protocols that support network slicing architecture establish wireless connections.

Wherein, one or more cameras can be set in both the master device 101 and the slave device 102 .

The master device 101 may use its own camera to collect image data (also referred to as a captured image), and the slave device 102 may also use its own camera to collect image data. Exemplarily, the slave device 102 can use its own camera to collect image data, and the slave device 102 can send the image data to the master device 101 , and the master device 101 displays the image data from the slave device 102 . Alternatively, the master device 101 and the slave device 102 can use their own cameras to capture image data at the same time, and the slave device 102 can send the image data to the master device 101, and the master device 101 simultaneously displays the images from the master device 101 and the slave device 102 data, so as to realize the distributed shooting function across devices.

Exemplarily, the above-mentioned master device 101 (or slave device 102 ) may specifically be a mobile phone, a tablet computer, a TV (also referred to as a smart TV, a smart screen, or a large-screen device), a notebook computer, and an Ultra-mobile personal computer (Ultra-mobile personal computer). Personal Computer, UMPC), handheld computers, netbooks, Personal Digital Assistant (PDA), wearable electronic devices (eg, smart watches, smart glasses, smart helmets, smart bracelets), in-vehicle devices, virtual reality devices, etc. For an electronic device with a photographing function, this embodiment of the present application does not impose any limitation on this.

Taking a mobile phone as the main device 101 as an example, an application with a shooting function, such as a camera application, a live broadcast application, or a video call application, can be installed in the mobile phone. As shown in FIG. 2 , taking the camera application as an example, after detecting that the user has opened the camera application, the mobile phone can turn on its own camera to start capturing images, and display the images in the preview frame 202 of the preview interface 201 in real time. Still taking the camera application as an example, still as shown in FIG. 2 , the mobile phone can set a preset button 203 in the preview interface 201 of the camera application, and the preset button 203 is used to trigger the mobile phone to acquire the shooting image captured by the slave device and display the image of the slave device. Take a picture. For example, the preset button 203 can trigger the mobile phone to display both its own and the captured images of the slave device in the preview interface 201 . For another example, the preset button 203 can trigger the mobile phone to switch its own shooting screen in the preview interface 201 to the shooting screen of the slave device. At this time, the mobile phone may not display its own shooting screen.

Exemplarily, after the mobile phone detects that the user clicks the preset button 203, the mobile phone can search for one or more electronic devices that currently have a photographing function. For example, whether each electronic device has a shooting function can be recorded in the server. Then, the mobile phone can query the server for an electronic device with a photographing function that is logged into the same account as the mobile phone (for example, a Huawei account).

Alternatively, the phone can search for electronic devices that are on the same Wi-Fi network as the phone. Further, the mobile phone can send a query request to each electronic device in the same Wi-Fi network, triggering the electronic device that receives the query request to send a response message to the mobile phone, and the response message can indicate whether it has a shooting function. In this way, the mobile phone can determine, according to the received response message, an electronic device with a photographing function in the current Wi-Fi network.

For example, the electronic devices with shooting function searched by the mobile phone include TV 1, watch 2 and tablet computer 3. The mobile phone can use these three electronic devices as slave devices to establish network connections with TV 1, watch 2 and tablet computer 3 respectively. For example, the mobile phone can establish a Wi-Fi connection or a Wi-Fi P2P connection with the TV 1, the watch 2 and the tablet computer 3 respectively, or the mobile phone can directly establish a mobile network connection with the TV 1, the watch 2 and the tablet computer 3. The mobile network Including but not limited to mobile networks supporting 2G, 3G, 4G, 5G and subsequent standard protocols.

Of course, the mobile phone can also automatically search for and connect to one or more electronic devices when the user opens the camera application or when the user triggers an option in the camera application, which is not limited in this embodiment of the present application.

As a possible implementation manner, an application for managing smart home devices in the home (such as a TV, an air conditioner, a speaker, or a refrigerator, etc.) may be installed in the mobile phone. Taking a smart home application as an example, the user can add one or more smart home devices in the smart home application, so that the smart home device added by the user is associated with the mobile phone. For example, a two-dimensional code containing device information such as device identification can be set on a smart home device. After the user scans the two-dimensional code with the smart home application of the mobile phone, the corresponding smart home device can be added to the smart home application, thereby establishing a smart home application. The relationship between home equipment and mobile phones. In this embodiment of the present application, when one or more smart home devices added in the smart home application go online, for example, when the mobile phone detects a Wi-Fi signal sent by the added smart home device, the mobile phone can communicate with the smart home device. Establish a network connection.

Exemplarily, if the mobile phone establishes a network connection with the TV 1 first, the mobile phone can send a shooting instruction to the TV 1, so that the TV 1 can use its own camera to start capturing shooting images in response to the shooting instruction. Moreover, as shown in FIG. 3 , the TV 1 can send the captured image 301 captured in real time to the mobile phone. At this time, the mobile phone can display the shooting screen 301 from the TV 1 in the preview interface 201 of the camera application. In this way, the mobile phone can switch the shooting screen of a certain slave device (eg, TV 1 ) to be displayed in the mobile phone (ie, the master device) in response to the user's operation of clicking the preset button 203 . Of course, after the mobile phone establishes a network connection with watch 2 (or tablet computer 3), the mobile phone can also instruct watch 2 (or tablet computer 3) to start capturing images and send the captured images to the phone, but the phone may not display the images. Shots from watch 2 (or tablet 3).

Subsequently, the TV 1 , the watch 2 and the tablet computer 3 can detect in real time whether there is an audio signal input by the user while using the camera to capture and photographed images. For example, the TV 1, the watch 2 and the tablet 3 can use their own microphones to detect whether there is an audio signal input by the user. Exemplarily, if the tablet computer 3 detects that the user inputs an audio signal to itself, it means that the user using the tablet computer 3 is making a sound. At this time, as shown in FIG. 4 , the tablet computer 3 can add a preset sounding mark to the captured shooting picture 401 , and send the shooting picture 401 carrying the sounding mark to the mobile phone (ie, the main device). Furthermore, after the mobile phone receives the shooting screen 401 sent by the tablet computer 3, if it recognizes that a voice mark is set in the shooting screen 401, it means that the user using the tablet computer 3 is speaking at this time. Furthermore, as still shown in FIG. 4 , the mobile phone can display the shooting screen 401 from the tablet computer 3 in the preview interface 201 of the camera application. At this time, the mobile phone can stop displaying the shooting picture sent by the TV 1 in the preview interface 201 .

That is to say, in a distributed shooting scenario, multiple slave devices connected to a master device (eg, the above-mentioned mobile phone) can detect in real time whether a user inputs an audio signal to itself. When a user inputs an audio signal to a certain slave device (for example, the above-mentioned tablet computer 3 ), the slave device can actively add a preset sounding mark to the captured picture. In this way, when the master device receives the shooting picture sent by the slave device, it can know that the user inputs an audio signal to the slave device through the sounding mark in the shooting picture. Furthermore, the master device can display the shooting picture carrying the sounding mark on the display interface. In this way, when different users use their respective slave devices to input audio signals, the master device can flexibly and accurately switch the captured image from the slave device with the user's voice input to the master device for display, so that the user can focus on The shooting screen of the slave device input by the user's voice improves the user's experience.

In addition, the above-mentioned tablet computer 3 can not only send the shooting picture carrying the sound mark to the mobile phone (ie the master device), but also can send the shooting picture carrying the sound mark to other slave devices (for example, the above-mentioned TV 1 and/or watch. 2) In this way, other slave devices can also display the shooting picture carrying the sound mark on the display interface, so that users in the distributed shooting scene can focus on the shooting picture of the slave device with the user's voice input.

In other embodiments, after the master device (for example, the above-mentioned mobile phone) establishes a connection with each slave device, it can also continue to use its own camera to capture and take pictures. At this time, the mobile phone can set two windows in the above-mentioned preview interface 201, one of which is used to display the captured image sent from the device (for example, the above-mentioned tablet computer 3) with the user's voice according to the above method, and the other window is used to display the user's voice. The shooting images collected by the mobile phone (ie, the main device) are not limited in this embodiment of the present application.

Illustratively, taking a mobile phone as the main device 201 in the above-mentioned distributed shooting system 200 as an example, FIG. 5 shows a schematic structural diagram of the mobile phone.

The mobile phone may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, Speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, etc.

It can be understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile phone. In other embodiments of the present application, the mobile phone may include more or less components than shown, or some components may be combined, or some components may be separated, or different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 110 . If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby increasing the efficiency of the system.

The wireless communication function of the mobile phone can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, the baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G etc. applied on the mobile phone. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the same device as at least part of the modules of the processor 110 .

The wireless communication module 160 can provide applications on the mobile phone including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system ( global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.

In some embodiments, the antenna 1 of the mobile phone is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the mobile phone can communicate with the network and other devices through wireless communication technology.

The mobile phone realizes the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, the handset may include 1 or N display screens 194, where N is a positive integer greater than 1.

The mobile phone can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

Camera 193 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the mobile phone may include 1 or N cameras 193 , where N is a positive integer greater than 1.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone. The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example to save files like music, video etc in external memory card.

Internal memory 121 may be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the mobile phone by executing the instructions stored in the internal memory 121 . The internal memory 121 may include a storage program area and a storage data area. The storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area can store data (such as audio data, phone book, etc.) created during the use of the mobile phone. In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like.

The mobile phone can implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, and an application processor. Such as music playback, recording, etc.

The audio module 170 is used for converting digital audio information into analog audio signal output, and also for converting analog audio input into digital audio signal. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .

Speaker 170A, also referred to as a "speaker", is used to convert audio electrical signals into sound signals. The mobile phone can listen to music through the speaker 170A, or listen to hands-free calls.

The receiver 170B, also referred to as "earpiece", is used to convert audio electrical signals into sound signals. When the mobile phone receives a call or a voice message, the voice can be received by placing the receiver 170B close to the human ear.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C through a human mouth, and input the sound signal into the microphone 170C. The mobile phone may be provided with at least one microphone 170C. In other embodiments, the mobile phone may be provided with two microphones 170C, which in addition to collecting sound signals, may also implement a noise reduction function. In other embodiments, the mobile phone can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions.

The earphone jack 170D is used to connect wired earphones. The earphone interface 170D can be the USB interface 130, or can be a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

Of course, the mobile phone may also include a charging management module, a power management module, a battery, a button, an indicator, and one or more SIM card interfaces, which are not limited in this embodiment of the present application.

The software system of the above mobile phone may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take an Android system with a layered architecture as an example to exemplarily describe the software structure of a mobile phone. Of course, in other operating systems (such as Hongmeng system, Linux system, etc.), as long as the functions implemented by each functional module are similar to the embodiments of the present application, it falls within the scope of the claims of the present application and their equivalent technologies.

FIG. 6 is a block diagram of a software structure of a mobile phone provided by an embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the HAL (hardware abstraction layer, hardware abstraction layer) layer and kernel layer.

The application layer can include a series of application packages.

As shown in FIG. 6 , applications such as call, memo, browser, contact, gallery, calendar, map, Bluetooth, music, video, and short message can be installed in the application layer.

In this embodiment of the present application, an application with a shooting function may be installed in the application layer, for example, a camera application, a video call application, or a live broadcast application. Of course, when other applications need to use the shooting function, an application with a shooting function such as a camera application can also be invoked to realize the shooting function.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in FIG. 6 , taking a camera application as an example, a camera service (CameraService) is set in the application framework layer. The camera application can start CameraService by calling the preset API. CameraService can interact with the camera HAL (Camera HAL) in the HAL (hardware abstraction layer, hardware abstraction layer) during the running process. Among them, the Camera HAL is responsible for interacting with the hardware devices (such as cameras) that realize the shooting function in the mobile phone. On the one hand, the Camera HAL hides the implementation details of the relevant hardware devices (such as specific image processing algorithms), and on the other hand can provide the Android system. Call the interface of the relevant hardware device.

Exemplarily, the camera application may send relevant control instructions (such as preview, zoom in, photographing or video recording instructions) issued by the user to the CameraService. On the one hand, CameraService can send received control commands to Camera HAL, so that Camera HAL can call the camera driver in the kernel layer according to the received control commands, and drive hardware devices such as cameras to collect raw image data in response to the control commands. For example, the camera can transmit each frame of raw image data collected to the Camera HAL through the camera driver at a certain frame rate. For the transfer process of the control instruction inside the operating system, reference may be made to the specific transfer process of the control flow in FIG. 6 .

On the other hand, after the CameraService receives the above control instruction, it can determine the shooting strategy at this time according to the received control instruction, and the specific image processing task that needs to be performed on the collected image data is set in the shooting strategy. For example, in preview mode, CameraService can set image processing task 1 in the shooting strategy to implement the face detection function. For another example, if the user enables the beauty function in the preview mode, the CameraService may also set image processing task 2 in the shooting strategy to implement the beauty function. Furthermore, CameraService can send the determined shooting strategy to Camera HAL.

When the Camera HAL receives each frame of image data collected by the camera, it can perform corresponding image processing tasks on the above image data according to the shooting strategy issued by CameraService, and obtain each frame of image processing after image processing. For example, Camera HAL can perform image processing task 1 for each frame of image data received according to shooting strategy 1, and obtain each corresponding frame of shooting picture. After the shooting strategy 1 is updated to the shooting strategy 2, the Camera HAL can perform image processing task 2 and image processing task 3 for each frame of image data received according to the shooting strategy 2, and obtain the corresponding shooting picture of each frame.

Subsequently, Camera HAL can report each frame of captured image after image processing to the camera application through CameraService, and the camera application can display each frame of captured image on the display interface, or the camera application can display each frame of captured image in the form of photos or videos. A frame shot is saved in the phone. For the transfer process of the above-mentioned shooting images inside the operating system, reference may be made to the specific transfer process of the data flow in FIG. 6 .

In addition, the application framework layer may also include a window manager, a content provider, a view system, a resource manager, a notification manager, and the like, which are not limited in this embodiment of the present application.

For example, the above-mentioned window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc. The above content providers are used to store and retrieve data and make these data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone book, etc. The view system described above can be used to build the display interface of an application. Each display interface can consist of one or more controls. Generally speaking, controls may include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other interface elements. The above resource managers provide various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on. The notification manager described above enables applications to display notification information in the status bar, which can be used to convey notification-type messages, and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also display notifications in the status bar at the top of the system in the form of graphs or scroll bar text, such as notifications of applications running in the background, and notifications that appear on the screen in the form of dialog windows. For example, prompting text information in the status bar, sounding, vibrating, flashing lights, etc.

As shown in Figure 6, the Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

The core library consists of two parts: one is the function functions that the java language needs to call, and the other is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, safety and exception management, and garbage collection.

A system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

Among them, the surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is located under the HAL and is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, a sensor driver, and the like, which are not limited in this embodiment of the present application.

Based on the software architecture of the Android system shown in FIG. 6 , in the embodiment of the present application, as shown in FIG. 7 , a device virtualization (DeviceVirtualization) application for realizing a distributed shooting function may be installed in the application layer of the mobile phone, Subsequent can be referred to as DV applications. The DV application can run in the mobile phone as a system application. Alternatively, the functions implemented by the DV application can also be resident and run in the mobile phone in the form of system services.

When the mobile phone needs to realize the distributed shooting function together with other electronic devices, the DV application in the mobile phone can establish a network connection with the mobile phone by using one or more electronic devices as slave devices of the mobile phone. Still as shown in FIG. 7 , after the DV application of the mobile phone establishes a network connection with the slave device, the DV application can obtain the shooting capability parameter of the slave device based on the network connection, and the shooting capability parameter is used to indicate the shooting capability of the slave device. For example, the shooting capability parameter may include specific image processing algorithms supported by the slave device, relevant hardware parameters of the camera in the slave device, and the like. Furthermore, the DV application can call the preset interface of the HAL, and input the acquired shooting capability parameters into the preset interface, so as to create a HAL corresponding to the slave device in the HAL.

In the embodiment of the present application, the HAL created by the DV application according to the shooting capability parameters of the slave device may be referred to as the DMSDP (Distributed Mobile Sensing Development Platform) HAL, and may also be referred to as the virtual Camera HAL. Different from the traditional Camera HAL in the mobile phone, the DMSDP HAL does not correspond to the actual hardware device of the mobile phone, but corresponds to the slave device currently connected to the mobile phone. The mobile phone can be used as the master device to send and receive data with the slave device through the DMSDP HAL, and the slave device can be used as a virtual device of the mobile phone to cooperate with the slave device to complete various services in the distributed shooting scene.

Still as shown in Figure 7, in addition to creating the corresponding DMSDP HAL for the slave device of the mobile phone in the HAL, the DV application can also send the shooting capability parameters of the slave device to CameraService for saving, that is, register the current slave device in CameraService. shooting ability.

Next, taking the camera application as an example, when the mobile phone runs the camera application, CameraService can determine the shooting strategy in the shooting process in real time according to the control instructions (such as preview, zoom, video and other instructions) issued by the camera application, combined with the shooting capability of the slave device. For example, CameraService can set the image processing tasks that the mobile phone needs to perform and the image processing tasks that the slave device needs to perform in the shooting strategy according to the shooting capability parameters of the slave device. Furthermore, CameraService can send a shooting instruction corresponding to the shooting strategy to the slave device through the DMSDP HAL, and trigger the slave device to perform the corresponding image processing task.

In this way, when the mobile phone and the slave device realize the distributed shooting function, the mobile phone and the slave device can perform corresponding image processing on the image data according to their own shooting capabilities based on the above shooting strategy, so that the mobile phone and the slave device can be more efficient and flexible. The distributed shooting function can be realized by the coordination of the system, and the better shooting effect can be achieved in the distributed shooting scene.

In the embodiment of the present application, taking the mobile phone 1 as the master device as an example, the mobile phone 1 and multiple slave devices can perform synchronous shooting in a distributed shooting scene.

For example, if it is detected that the user opens the camera application of the mobile phone 1, the mobile phone 1 can open its own camera to start shooting, and, as shown in FIG. 8, the mobile phone 1 can display the captured shooting image in the preview interface 801 of the camera application . In this embodiment of the present application, the preview interface 801 may be provided with a preset button 802 for synchronizing shooting with the slave device, and the preset button 802 may be used to trigger the mobile phone 1 to display shooting images of two or more electronic devices. For example, the name of the preset button 802 may be "dual-view mode", "three-view mode" or "multi-view mode", and so on. Taking the "dual scene mode" as an example, after the mobile phone 1 detects that the user selects the preset button 802 named "dual scene mode", it can trigger the mobile phone 1 to create two windows in the preview interface 801, one of which is used to display the mobile phone 1 , and another window is used to display the shooting images of other electronic devices. It should be understood that if the mobile phone establishes a wireless connection with N electronic devices with cameras, when the user clicks the preset button 802 on the preview interface 801, the mobile phone 1 is triggered to create N+1 windows on the preview interface 801, and one window is used for The shooting pictures of the mobile phone 1 are displayed, and the other N windows are used to display the shooting pictures of N electronic devices. Of course, it is also possible to create fewer windows than N+1 or more windows than N+1 to display the shooting interface of mobile phone 1 and the shooting interfaces of other N electronic devices. If the number of windows created is less than N+1, some electronic devices can be selected. The camera interface of the device is not displayed. If more than N+1 windows are created, the more than N+1 windows may display other content, or repeatedly display the shooting interface of one or more electronic devices.

Of course, the mobile phone 1 can also set the above-mentioned preset button 802 for synchronous shooting with the slave device in the control center, pull-down menu, negative one-screen menu or other applications (such as video calling applications, live broadcasting applications, etc.) of the mobile phone 1, This embodiment of the present application does not impose any limitation on this.

In the following embodiments, the preset button 802 named "dual scene mode" is used as an example. When the mobile phone 1 detects that the user clicks the preset button 802, the DV application of the mobile phone 1 can trigger the mobile phone 1 to search for a nearby camera or camera with a shooting function. Multiple electronic devices. Of course, the mobile phone 1 can also automatically search for one or more nearby electronic devices with a shooting function after the user opens the camera application. Taking the DV application of mobile phone 1 detecting two electronic devices, TV 1 and mobile phone 2, for example, the DV application of mobile phone 1 can use TV 1 and mobile phone 2 as two slave devices of mobile phone 1 to establish network connection with mobile phone 1 respectively. For example, mobile phone 1 can establish a Wi-Fi P2P connection with TV 1, and mobile phone 1 can establish a Wi-Fi P2P connection with mobile phone 2. Furthermore, the DV application of the mobile phone 1 can acquire the shooting capability parameter 1 of the TV 1 based on the network connection with the TV 1 . Moreover, the DV application of the mobile phone 1 can acquire the shooting capability parameter 2 of the mobile phone 2 based on the network connection with the mobile phone 2 .

Furthermore, as shown in FIG. 9 , the DV application of the mobile phone 1 can create a corresponding DMSDP HAL in the HAL according to the acquired shooting capability parameter 1 and shooting capability parameter 2. For example, the DV application of the mobile phone 1 can create a virtual HAL 1 corresponding to the TV 1 according to the shooting capability parameter 1, and the DV application of the mobile phone 1 can create a virtual HAL 2 corresponding to the mobile phone 2 according to the shooting capability parameter 2. Virtual HAL 1 and Virtual HAL 2 can run as two functional modules in the DMSDP HAL. Subsequently, the mobile phone 1 can perform data interaction with the TV 1 through the virtual HAL 1, and the mobile phone 1 can perform data interaction with the mobile phone 2 through the virtual HAL 2.

It can be understood that when the mobile phone 1 detects a new slave device, it can also create a new virtual HAL in the DMSDP HAL according to the above method. Alternatively, when the mobile phone 1 detects that the network connection is disconnected from a current slave device, the corresponding virtual HAL can also be deleted in the DMSDP HAL. Alternatively, when the mobile phone 1 detects that the shooting capability parameter of a slave device is updated, the DV application of the mobile phone 1 can update the corresponding virtual HAL according to the updated shooting capability parameter, which is not limited in this embodiment of the present application.

In other embodiments, after the DV application of mobile phone 1 obtains the shooting capability parameters of TV 1 and mobile phone 2, it can also create two DMSDP HALs in the HAL, one of which corresponds to TV 1, and the other DMSDP HAL corresponds to Corresponding to the mobile phone 2, this embodiment of the present application does not impose any limitation on this.

Alternatively, after the mobile phone 1 searches for one or more electronic devices with a photographing function nearby, it can display a corresponding device list. In this way, the user can select one or more electronic devices that are the slave devices of the mobile phone 1 this time in the device list displayed by the mobile phone 1 . Subsequently, the mobile phone 1 can respond to the user's selection operation, establish a network connection with the corresponding electronic device according to the above method, and create a corresponding DMSDP HAL, which is not limited in this embodiment of the present application.

Still as shown in FIG. 9 , a Camera HAL corresponding to the camera of the mobile phone can also be set in the HAL of the mobile phone 1 . After detecting that the user has opened the camera application, the mobile phone 1 can open its own camera to collect image data. The camera of the mobile phone 1 can report the collected image data to the Camera HAL, and the Camera HAL processes the collected image data into each corresponding frame of shooting picture (ie, the shooting picture 1). Furthermore, the Camera HAL can report the obtained shooting picture 1 to the CameraService, and the CameraService reports the shooting picture 1 to the camera application, so that the camera application displays the shooting picture 1 on the display interface of the mobile phone 1.

In the embodiment of the present application, after the camera application of the mobile phone 1 detects that the user clicks the preset button 802, as shown in FIG. 10, the camera application can create a window 1201 and a window 1202 in the display interface. At this time, similar to the processing process of mobile phone 1 in FIG. 9 , the CameraService of mobile phone 1 can receive the shooting picture 1 from mobile phone 1 reported in real time by the Camera HAL, and the CameraService of mobile phone 1 can report the shooting picture 1 of mobile phone 1 to the camera. application. At this time, the camera application can display the photographed screen 1 of the mobile phone 1 in the window 1201 (or the window 1202 ).

At the same time, after the mobile phone 1 establishes a network connection with the current slave device, the camera application of the mobile phone 1 can also display the captured image from the slave device in real time in another window (eg, window 1202 ) of the display interface. For example, the mobile phone 1 can display the photographed screen of the first slave device that establishes a network connection with the mobile phone 1 in the window 1202 by default.

Still taking the slave devices of mobile phone 1 as TV 1 and mobile phone 2 as an example, when mobile phone 1 first establishes a network connection with TV 1, and creates a corresponding virtual HAL 1 in the DMSDP HAL according to the shooting capability parameters of TV 1, mobile phone 1 can send The television 1 sends a shooting instruction 1 . Still as shown in FIG. 9 , the television 1 can use the camera of the television 1 to acquire the photographed image 2 in real time in response to the photographing instruction 1 . Furthermore, the TV 1 can send the captured image 2 to the virtual HAL 1 of the mobile phone 1. After the virtual HAL 1 receives the captured image 2 from the TV 1, it can report the captured image 2 to the camera application through the CameraService of the mobile phone 1. Then, as shown in FIG. 10 , the camera application can display the shot 2 from the TV 1 in the window 1202 . In this way, when the user enables the synchronous shooting function of "dual scene mode" in the mobile phone 1, the mobile phone 1 can synchronously display the shooting screen of the mobile phone itself and the shooting screen of the first slave device connected to the mobile phone 1 on the display of the mobile phone 1. in the interface.

Of course, the mobile phone 1 can also be set to display the captured image of the second (or Nth) slave device connected to the mobile phone 1 in the window 1202 . Alternatively, the mobile phone 1 can also be set to display the shooting screen of a specified slave device in the window 1202 . Alternatively, the mobile phone 1 may also display the shooting images of multiple slave devices connected to the mobile phone 1 on the display interface. Of course, the mobile phone 1 may also not display any captured images of the slave device in the window 1202, which is not limited in this embodiment of the present application.

On the other hand, after mobile phone 1 establishes a network connection with TV 1, when mobile phone 1 establishes a network connection with mobile phone 2, and creates a corresponding virtual HAL 2 in the DMSDP HAL according to the shooting capability parameters of mobile phone 2, because mobile phone 2 is not The first slave device that establishes a network connection with the mobile phone 1 , that is, the window 1202 does not need to display the shooting screen of the mobile phone 2 , so the mobile phone 1 may not send a shooting instruction to the mobile phone 2 . In this way, the mobile phone 2 will not turn on the camera in response to the shooting instruction sent by the mobile phone 1 to obtain the corresponding shooting picture in real time, and the mobile phone 1 will not receive the shooting picture from the mobile phone 2 .

Alternatively, in other embodiments, as shown in FIG. 11 , a control module 1 may also be set in the DMSDP HAL of the mobile phone 1, and the control module 1 may be used to determine which slave device to send the captured image from to CameraService. For example, when the DV application of the mobile phone 1 creates a virtual HAL corresponding to each slave device, the corresponding relationship between different slave devices and different virtual HALs can be saved in the control module 1 of the DMSDP HAL. As shown in Table 1, TV 1 and mobile phone 2 are two slave devices currently connected to mobile phone 1. Among them, TV 1 corresponds to virtual HAL 1, TV 1 is provided with a camera A, and the shooting capability parameter of camera A is parameter 1. The mobile phone 2 corresponds to the virtual HAL 2. The mobile phone 2 is provided with two cameras (ie, camera B and camera C). The shooting capability parameter of camera B is parameter 2, and the shooting capability parameter of camera C is parameter 3.

Table 1

Then, if the mobile phone 1 establishes a network connection with the TV 1 first, the control module 1 can set the corresponding relationship between the window 1202 and the TV 1, that is, the window 1202 needs to display the shooting image of the TV 1. Further, the mobile phone 1 can send a shooting instruction to the TV 1 according to the above method, instructing the TV 1 to send the shooting picture 2 to the virtual HAL 1 of the mobile phone 1. As shown in FIG. 11 , after receiving the shooting picture 2 from the TV 1, the virtual HAL 1 of the mobile phone 1 can send the shooting picture 2 to the above-mentioned control module 1, because the control module 1 has already set the connection between the window 1202 and the TV 1. Therefore, after the control module 1 recognizes that the shooting picture 2 is from the TV 1, it can report the shooting picture 2 to the camera application through CameraService, and the camera application can display the shooting picture 2 from the TV 1 in the window 1202. .

And, still as shown in Figure 11, when the mobile phone 1 and the mobile phone 2 establish a network connection, and create a corresponding virtual HAL 2 in the DMSDP HAL, the mobile phone 1 can also send a shooting instruction (that is, the shooting instruction 2) to the mobile phone 2. The mobile phone 2 can use the camera of the mobile phone 2 to acquire the corresponding shooting image 3 in real time in response to the shooting instruction 2 . Furthermore, the mobile phone 2 can send the captured image 3 to the virtual HAL 2 of the mobile phone 1 . After the virtual HAL 2 receives the shooting picture 3 from the mobile phone 2, the shooting picture 3 can be sent to the above-mentioned control module 1, because the control module 1 has set the corresponding relationship between the window 1202 and the TV 1, therefore, when the control module 1 When recognizing that the shooting picture 3 is not the shooting picture from the TV 1, the control module 1 will not report the shooting picture 3 to CameraService. For example, the control module 1 can discard or delete the photographed image 3 sent by the mobile phone 2 . In this way, the camera application will not receive the shooting image 3 from the mobile phone 2 , nor will it display the shooting image 3 on the display interface of the mobile phone 1 .

Exemplarily, when each slave device (eg, the above-mentioned TV 1 and mobile phone 2 ) sends its own shooting picture to the master device (such as the above-mentioned mobile phone 1 ), the shooting picture can be sent to the mobile phone 1 in the form of a data packet.

As shown in FIG. 12 , the data packet of the above-mentioned captured picture generally includes a header (head) and a data (data) part. Wherein, the data part includes specific image data in the shooting picture, such as YUV (Y is used to indicate brightness, and U and V are used to indicate chromaticity) data. The header (head) includes parameters related to the data part, for example, the name of the captured picture, the capturing time, codec information, and the like. In this embodiment of the present application, the slave device may carry its own device identifier in the header of the data packet. For example, the TV 1 can carry the device ID of the TV 1 in the header of the data packet corresponding to the shooting picture 2 , and the mobile phone 2 can carry the device ID of the mobile phone 2 in the header of the data packet corresponding to the shooting picture 3 . In this way, the control module 1 of the DMSDP HAL in the mobile phone 1 can identify which slave device the captured image in the currently received data packet is from by parsing the packet header of the data packet.

Alternatively, the television 1 may also carry the camera identifier of the camera used for shooting the picture 2 in the header of the corresponding data packet. In this way, the control module 1 of the DMSDP HAL in the mobile phone 1 can also know the specific camera used when the TV 1 shoots the picture 2 by parsing the packet header of the data packet.

Of course, the control module 1 of the DMSDP HAL in the mobile phone 1 can also determine the shooting picture obtained from the virtual HAL 1 as the shooting picture of the TV 1. Correspondingly, the shooting picture obtained from the virtual HAL 2 can be determined as the mobile phone. 2 of the shooting screen. Alternatively, since the network connection 1 between the mobile phone 1 and the TV 1 may be different from the network connection 2 between the mobile phone 1 and the mobile phone 2, the control module 1 of the DMSDP HAL in the mobile phone 1 can obtain the captured image from the network connection 1. It is determined as the shooting image of the TV 1 , and the shooting image obtained from the network connection 2 is determined as the shooting image of the mobile phone 2 .

In the embodiment of the present application, the TV 1 and the mobile phone 2 are still taken as the slave devices of the mobile phone 1 as an example. When the TV 1 and the mobile phone 2 use their own cameras to capture the captured images and send the captured images to the mobile phone 1, they can also use the The microphone collects the audio signal input by the user, and then determines whether the user is using the slave device to utter based on the collected audio signal.

Taking mobile phone 2 as an example, as shown in FIG. 13 , the architecture of the operating system in mobile phone 2 is similar to that of the operating system in mobile phone 1 . The application layer of the mobile phone 2 may install a proxy application, and the proxy application is used to send and receive data with other devices (eg, the mobile phone 1 or the TV 1 ). Alternatively, the proxy application may also run in the mobile phone 2 in the form of an SDK (Software Development Kit, software development kit) or a system service. CameraService can also be set in the application framework layer of mobile phone 2. Camera HAL can also be set in the HAL of mobile phone 2. The Camera HAL of mobile phone 2 corresponds to the camera of mobile phone 2.

After the mobile phone 2 establishes a network connection with the mobile phone 1 , the proxy application of the mobile phone 2 can receive the shooting instruction sent by the mobile phone 1 . In response to the shooting instruction, the proxy application of the mobile phone 2 can call the camera of the mobile phone 2 through its own CameraService and Camera HAL to collect the corresponding shooting picture (for example, the above-mentioned shooting picture 3). The Camera HAL of the mobile phone 2 can receive the shooting picture 3 sent by the camera in real time, and then the Camera HAL of the mobile phone 2 can send the received shooting picture 3 to the proxy application through CameraService, and the proxy application sends the shooting picture 3 of the mobile phone 2. To the virtual HAL 2 corresponding to the mobile phone 2 in the mobile phone 1, so that the mobile phone 1 can obtain the shooting picture of the mobile phone 2 in real time. Of course, after the Camera HAL of the mobile phone 2 receives the shooting picture 3, it can also perform related image processing (such as beautifying, focusing, etc.) on the shooting picture 3, and then send the shooting picture after image processing to the agent application. The embodiment does not make any limitation on this.

Still as shown in FIG. 13 , the operating system of the mobile phone 2 may further include related modules for implementing audio functions. For example, a recording (AudioRecord) and an audio manager (AudioFlinger) can be set in the application framework layer of the mobile phone 2, and an audio HAL (Audio HAL) can be set in the HAL of the mobile phone 2. The Audio HAL corresponds to audio input and output devices such as the microphone and speaker of the mobile phone 2 . For example, the microphone of the mobile phone 2 can report the collected audio signal to AudioFlinger. AudioFlinger is used to mix, resample, and set sound effects on the collected audio signals. AudioRecord is used to obtain the corresponding audio signal from AudioFlinger for audio recording. AudioRecord and CameraService can interact with proxy applications through a unified interface (such as the HwMediaProject interface). For example, AudioRecord can report the recorded audio signal to the proxy application through the HwMediaProject interface, and CameraService can report the captured image to the proxy application through the HwMediaProject interface.

In the embodiment of the present application, in the process that the mobile phone 2 uses the camera to capture the captured image and sends the captured image to the mobile phone 1 , the mobile phone 2 can also detect whether the user inputs an audio signal to the mobile phone 2 through its own microphone. Still as shown in Figure 13, when the microphone of the mobile phone 2 detects the audio signal, it can send the detected audio signal to the Audio HAL of the mobile phone 2, and the Audio HAL of the mobile phone 2 can send the received audio signal to AudioFlinger. Furthermore, AudioRecord can obtain the audio signal from AudioFlinger, and report the audio signal to the proxy application through the HwMediaProject interface. After receiving the audio signal detected by the microphone, the proxy application can determine whether the audio signal is the sound input by the user to the mobile phone 2 .

Exemplarily, the mobile phone 2 may store voiceprint features of one or more legitimate users who use the mobile phone 2 . After the proxy application of the mobile phone 2 obtains the audio signal detected by the current microphone, the voiceprint feature in the audio signal can be extracted. Furthermore, the proxy application of the mobile phone 2 can compare whether the extracted voiceprint feature is the voiceprint feature of a legitimate user. If it is the voiceprint feature of a legitimate user, the proxy application of the mobile phone 2 can determine that the currently detected audio signal is the sound input by the user to the mobile phone 2 . Otherwise, the proxy application of the mobile phone 2 may determine that the currently detected audio signal is not the sound input by the user to the mobile phone 2, and the audio signal may be noise or other sounds input by the user to other devices.

Or, still as shown in FIG. 13 , the proxy application of the mobile phone 2 can also receive the captured image reported by CameraService while receiving the audio signal reported by AudioRecord. Then, the proxy application of the mobile phone 2 can determine whether the currently received audio signal is the sound input by the user to the mobile phone 2 in combination with the received photographed image and the audio signal. For example, as shown in FIG. 14 , for the audio signal and the captured image received at the same time, the proxy application can use a face recognition algorithm to detect whether the corresponding captured image includes a face image. If a face image is included, the proxy application can further extract lip-shape features in the face image. Furthermore, the proxy application can determine whether a user inputs a sound to the mobile phone 2 by comparing whether the audio signal at the same moment matches the mouth shape feature. For example, if the agent application detects that the audio signal within 500ms matches the corresponding mouth feature, the agent application can determine that the currently detected audio signal is the sound input by the user to the mobile phone 2, that is, the user using the mobile phone 2 is making a sound.

Alternatively, in addition to comparing whether the audio signal at the same moment matches the mouth shape feature, the proxy application can also detect whether the loudness of the audio signal is greater than a preset value. For example, if the agent application detects that the audio signal within 500ms of continuous time matches the corresponding mouth shape feature, and the loudness of the audio signal within the 500ms is greater than the preset value, the agent application can determine that the currently detected audio signal is the audio signal sent by the user to the mobile phone. 2 Input sounds.

Of course, the mobile phone 2 can also determine whether the currently detected audio signal is the sound input to the mobile phone 2 by the user through other methods. For example, the mobile phone 2 may request the server to determine whether the currently detected audio signal is the sound input by the user to the mobile phone 2, which is not limited in this embodiment of the present application.

In the embodiment of the present application, if the proxy application of the mobile phone 2 determines that the currently received audio signal is the sound input by the user to the mobile phone 2, it means that among the multiple slave devices of the mobile phone 1, a user is using the mobile phone 2 to speak. At this time, as shown in FIG. 15 , the proxy application of the mobile phone 2 can add a preset sounding mark to the shooting picture (ie the shooting picture 3 ) reported by CameraService, and then send the shooting picture 3 with the sounding mark added to the mobile phone 1. Virtual HAL 2. For example, the above-mentioned sounding identification may be a specific mark bit, and the proxy application of the mobile phone 2 may add the above-mentioned sounding identification to the packet header of the data packet of the captured image 3 received in real time. Subsequently, when the proxy application of the mobile phone 2 determines that there is no audio signal input, or, when the proxy application of the mobile phone 2 determines that the currently received audio signal is not the sound input by the user to the mobile phone 2, the proxy application of the mobile phone 2 can stop sending the corresponding audio signal to the mobile phone 2. The above-mentioned sounding mark is added to the data packet of the shooting picture 3.

In this way, still as shown in Figure 15, after the virtual HAL 2 of the mobile phone 1 receives the data packet of the shooting screen 3 from the mobile phone 2, it can send the data packet of the shooting screen 3 to the control module 1 of the DMSDP HAL. Further, if the control module 1 of the DMSDP HAL parses out the sounding mark of the data packet, it is explained that the user is inputting the sound to the mobile phone 2 when shooting this shooting picture 3, then the control module 1 can modify the corresponding relationship between the window 1202 and the TV 1 is the corresponding relationship between the window 1202 and the mobile phone 2 . Further, the control module 1 of the DMSDP HAL can report the shooting picture 3 of the mobile phone 2 to the CameraService of the mobile phone 1, and stop reporting the shooting picture 2 of the TV 1 to the CameraService. That is to say, the control module 1 of the DMSDP HAL in the mobile phone 1 can determine the slave device that the user is making a sound by analyzing the sound-emitting identifier in the shooting screen, and then send the shooting screen of the slave device that the user is making a sound to the CameraService.

Exemplarily, taking the example that the packet header of the data packet 1 includes the utterance identifier, the control module 1 of the DMSDP HAL can send the image data in the data part of the data packet 1 to the CameraService of the mobile phone 1. Alternatively, the control module 1 of the DMSDP HAL can also send the image data of the data part and the voice identification to the CameraService of the mobile phone 1.

After the CameraService of the mobile phone 1 reports the shooting screen of the mobile phone 2 to the camera application, as shown in FIG. 16 , the camera application can display the received shooting screen 3 of the mobile phone 2 in the window 1202 . At this time, it was originally in the window 1202 The displayed shooting screen 2 of the television 1 is switched to the shooting screen 3 of the mobile phone 2 . In addition, the camera application can still display the shooting screen 1 shot by the mobile phone 1 itself in the window 1201 . That is to say, as the master device, the mobile phone 1 can display the shooting screen shot by the mobile phone 1 itself, and at the same time display the shooting screen of the slave device that the user is uttering. In some embodiments, still as shown in FIG. 16 , if the captured image 3 of the mobile phone 2 received by the camera application carries a sounding mark, the camera application can also display a first prompt message 1601 in the window 1202 for sending a notification to the camera application. The user prompts that the user using the cell phone 2 corresponding to the window 1202 is speaking. The first prompt message 1601 may be text, icon, animation, or voice, etc., which is not limited in this embodiment of the present application.

In addition, the mobile phone 2 can also send the detected audio signal to the mobile phone 1 in real time while sending the photographed image to the mobile phone 1 . Subsequently, while the mobile phone 1 displays the photographed image of the mobile phone 2 in the window 1202 , it can also play the audio received by the mobile phone 2 . In this way, in scenarios such as a conference or a video call, as the master device, the mobile phone 1 can automatically present to the user the captured images and audio content of the slave device used by the speaking user.

In some embodiments, if the proxy application of the mobile phone 2 determines that the currently received audio signal is not the sound input by the user to the mobile phone 2, or, if the mobile phone 2 does not detect an audio signal, the mobile phone 2 does not need to record the captured image. Add sound mark to . Further, after the control module 1 of the DMSDP HAL in the mobile phone 1 receives the shooting picture sent by the mobile phone 2, it will not report the shooting picture of the mobile phone 2 to the CameraService of the mobile phone 1, and the camera application will not display the shooting picture of the mobile phone 2 on the display. in the interface.

In some embodiments, the TV 1, as a slave device of the mobile phone 1, can also detect whether a user inputs an audio signal to the TV 1 according to the above method. If the TV 1 detects that the user inputs an audio signal to the TV 1, the TV 1 can also send the shooting image with the sounding mark added to the mobile phone 1 according to the above method. If the mobile phone 1 recognizes that the sounding mark is added to the shooting picture of the TV 1, but the sounding mark is not added to the shooting picture of the mobile phone 2, the mobile phone 1 can continue to display the shooting picture of the TV 1 in the window 1202.

If the mobile phone 1 recognizes that the sounding mark is added to the shooting picture of the TV 1, and the sounding mark is also added to the shooting picture of the mobile phone 2, because the control module 1 of the DMSDP HAL in the mobile phone 1 has recorded the corresponding relationship between the TV 1 and the window 1202 , therefore, the control module 1 of the DMSDP HAL can continue to report the shooting picture of the TV 1 to the CameraService of the mobile phone 1, without reporting the shooting picture of the mobile phone 2 to the CameraService of the mobile phone 1, so that the camera application can continue to display the picture of the TV 1 in the window 1202. Take a picture.

Or, if the mobile phone 1 recognizes that a sounding mark is added to the shooting picture of the TV 1, and that the sounding mark is also added to the shooting picture of the mobile phone 2, then the control module 1 of the DMSDP HAL in the mobile phone 1 can record these two slave devices. All images are reported to CameraService. At this time, after receiving the shooting image of the TV 1 and the shooting image of the mobile phone 2 , the camera application can display the shooting image of the TV 1 in the window 1202 . Moreover, the camera application can create a new window in the current display interface, and display the shooting image of the mobile phone 2 in the newly created window. In this way, the shooting interface of each slave device that detects the voice of the user can be displayed on the display interface of the master device.

In some embodiments, still taking the mobile phone 1 to switch the shooting screen of the TV 1 in the window 1202 to the shooting screen of the mobile phone 2 as an example, the control module 1 of the DMSDP HAL in the mobile phone 1 updates the corresponding relationship between the window 1202 and the TV 1 to the window 1202 After the corresponding relationship with the mobile phone 2, the current time can be recorded as the time when the window 1202 performs screen switching. Subsequently, if the control module 1 of the DMSDP HAL in the mobile phone 1 receives a shooting picture with a sounding mark sent from other devices (for example, TV 1), the control module 1 of the DMSDP HAL can determine the distance from the last switch in the window 1202 Whether the time to capture the picture exceeds the preset time (for example, 3 seconds). If the time since the last time the shooting screen was switched in the window 1202 has not exceeded the preset time, it means that the mobile phone 1 has switched the shooting screen in the window 1202 for a short time. For user experience, the mobile phone 1 can continue to display the shooting screen of the mobile phone 2 in the window 1202 . If the time since the last time the shooting screen was switched in the window 1202 exceeds the preset time, the mobile phone 1 can switch the shooting screen of the mobile phone 2 displayed in the window 1202 to the currently received shooting screen carrying the sounding logo.

It can be seen that in the above distributed shooting scenario, after the mobile phone 1 (ie the master device) establishes a network connection with each slave device (for example, the above-mentioned TV 1 and mobile phone 2), if a slave device detects that there is a user inputting audio to itself. signal, the slave device can notify the master device to display the shooting screen of the slave device by adding a sound mark to the shooting screen. In this way, when different users use their respective slave devices to input audio signals, the master device can flexibly and accurately switch the shooting image captured by the slave device with the user's voice input to the master device for display, so that the user can focus on the The shooting screen of the slave device with the user's voice input improves the user's experience.

Of course, after the slave device detects that a user has input an audio signal to itself, the slave device can also notify the master device to display the shooting picture of the slave device in other ways, which is not limited in this embodiment of the present application. For example, after mobile phone 2 detects that the currently received audio signal is the sound input by the user to mobile phone 2, mobile phone 2 can send a first indication message to mobile phone 1, and the first indication message is used to indicate that the user using mobile phone 2 is making a sound. Furthermore, after receiving the first instruction message sent by the mobile phone 2, the mobile phone 1 can display the subsequently received photographed images of the mobile phone 2 on the display interface. Subsequently, when the mobile phone 2 does not receive an audio signal within a preset time, the mobile phone 2 may send a second instruction message to the mobile phone 1, and the second instruction message is used to instruct the user using the mobile phone 2 to stop speaking. Furthermore, after the mobile phone 1 receives the second instruction message sent by the mobile phone 2, it can stop displaying the shooting screen of the mobile phone 2 on the display interface. Alternatively, if the mobile phone 1 receives the first instruction message sent by other slave devices when displaying the shooting screen of the mobile phone 2, the mobile phone 1 can also display the shooting screen of other slave devices and stop displaying the shooting screen of the mobile phone 2.

In some embodiments, in addition to the mobile phone 1 (that is, the master device) that can display the shooting screen of the slave device (such as the above-mentioned mobile phone 2 ) with the user's voice input in the display interface, other slave devices (such as the TV 1 ) can also display The interface displays the shooting screen of the slave device input by the user's voice.

Still taking the mobile phone 2 detecting a user inputting an audio signal to the mobile phone 2 as an example, the mobile phone 2 can not only send the shooting picture carrying the sounding mark to the mobile phone 1, but also send the shooting picture carrying the sounding mark to the TV 1. As shown in FIG. 17 , the architecture of the operating system in the TV 1 is similar to the architecture of the operating system in the mobile phone 2 . The application layer of the TV 1 may install a proxy application, and the proxy application is used to send and receive data with other devices (eg, the mobile phone 1 or the mobile phone 2 ). CameraService can also be set in the application framework layer of TV 1. The Camera HAL can also be set in the HAL of TV 1. The Camera HAL of TV 1 corresponds to the camera of TV 1. In addition, AudioRecord and AudioFlinger can also be set in the application framework layer of TV 1, and Audio HAL can also be set in the HAL of TV 1. The Audio HAL corresponds to audio input and output devices such as the microphone and speaker of the TV 1 .

Among them, the process of sending a captured image to the mobile phone 1 by the TV 1 as a slave device of the mobile phone 1 is similar to the process of sending the captured image from the mobile phone 2 to the mobile phone 1, and the process of the TV 1 detecting whether a user inputs an audio signal to the TV 1 is similar to the process of the mobile phone 2 The process of detecting whether a user inputs an audio signal to the mobile phone 2 is similar, so it is not repeated here.

The difference is that, as shown in FIG. 17 , the application layer of the TV 1 can also set a camera application and a DV application. The TV 1 can not only establish a network connection with the mobile phone 1, but also establish a network connection with the mobile phone 2. Moreover, the TV 1 can acquire the shooting capability parameters of the mobile phone 1 and the shooting capability parameters of the mobile phone 2 . For example, the mobile phone 1 can send the shooting capability parameters of the TV 1 and the shooting capability parameters of the mobile phone 2 obtained by the mobile phone 1 to the DV application of the TV 1 . Further, the DV application of the TV 1 may create a corresponding DMSDP HAL in the HAL of the TV 1. For example, the DV application of TV 1 can create a virtual HAL 3 corresponding to mobile phone 1 and a virtual HAL 4 corresponding to mobile phone 2 in the DMSDP HAL. Subsequently, the TV 1 can receive the data sent by the mobile phone 1 through the virtual HAL 3, and can also receive the data sent by the mobile phone 2 through the virtual HAL 4. Moreover, similar to the control module 1 in the DMSDP HAL of the mobile phone 1, a control module (hereinafter referred to as the control module 2) can also be set in the DMSDP HAL of the TV 1, and the control module 2 is used to determine which camera service to send to the TV 1 comes from. The shot of the device.

Subsequently, as shown in FIG. 17 , if another slave device (such as mobile phone 2) of mobile phone 1 detects that a user inputs an audio signal to mobile phone 2, mobile phone 2 can also record the shooting picture (such as the above-mentioned shooting Picture 3) is sent to the virtual HAL 4 of the TV 1. Further, the virtual HAL 4 can report the received shooting picture 3 to the control module 2. Similar to the working principle of the above-mentioned control module 1, after the control module 2 receives the shooting picture 3 from the mobile phone 2, if the sounding identification carried by the shooting picture is parsed, the control module 2 can report the shooting picture 3 of the mobile phone 2 to the TV. 1 CameraService. Subsequently, the CameraService of the TV 1 can report the shooting screen 3 of the mobile phone 2 to the camera application of the TV 1 , and the camera application of the TV 1 displays the shooting screen 3 of the mobile phone 2 on the display interface of the TV 1 . In this way, as a slave device of the mobile phone 1 , the television 1 can also present to the user the shooting screen of the slave device (for example, the above-mentioned mobile phone 2 ) to which the user's voice is input.

Exemplarily, as shown in (a) of FIG. 18 , after receiving the shooting screen 3 of the mobile phone 2 reported by CameraService, the camera application of the TV 1 can display only the shooting screen 3 of the mobile phone 2 in the display interface. At this time, if the CameraService of TV 1 receives the shooting picture of TV 1 (for example, the shooting picture 2) reported by the Camera HAL of TV 1, the CameraService of TV 1 can report the shooting picture 2 of TV 1 to the proxy application of TV 1. The proxy application of the TV 1 sends the captured image 2 of the TV 1 to the mobile phone 1 . That is to say, the CameraService of the TV 1 does not need to report the shooting picture 2 of the TV 1 to the camera application of the TV 1, and the camera application of the TV 1 will not display its own shooting picture 2 on the display interface.

Or, in some scenarios, the CameraService of the TV 1 can receive the shooting picture 2 of the TV 1 reported by the Camera HAL of the TV 1, and can also receive the shooting picture 3 of the mobile phone 2 reported by the DMSDP HAL of the TV 1. At this time, the CameraService of the TV 1 can report the captured images of the two devices to the camera application of the TV 1 . Furthermore, as shown in (b) of FIG. 18 , the camera application of TV 1 can also create two windows (ie, window 1801 and window 1802 ) in its display interface. At this time, the camera application of TV 1 can convert the phone 2 The shooting screen 3 of the TV 1 is displayed in one of the windows (eg, window 1801 ), and the camera application of the TV 1 can display the shooting screen 2 of the TV 1 itself in another window (eg, the window 1802 ). In this way, the television 1 may display a photographing screen shot by the television 1 itself and a photographing screen of a slave device input by the user's voice.

It should be noted that in the above embodiment, the slave devices of the mobile phone 1 include a TV 1 and a mobile phone 2. When there is a mobile phone 2 input by the user's voice, the mobile phone 2 can send the captured image of the mobile phone 2 to the mobile phone 1 and the TV 1 for display. . It can be understood that the mobile phone 2 can also receive and display the captured images of other slave devices (eg, the TV 1 ) according to the above method, which is not limited in this embodiment of the present application.

In other embodiments, the mobile phone 1, as the main device, uses its own camera to capture images while using a microphone to capture audio signals input by the user, and then determines whether the user is using the mobile phone 1 to make sounds based on the collected audio signals. The method for collecting audio signals by the mobile phone 1 and determining whether the user inputs the audio signals to the mobile phone 1 based on the collected audio signals can be referred to in the above-mentioned embodiment for collecting the audio signals by the mobile phone 2, and determining whether the user wants to input the audio signals based on the collected audio signals. The method for inputting the audio signal by the mobile phone 2 is omitted here.

Exemplarily, if the mobile phone 1 determines that the user is currently inputting an audio signal to the mobile phone 1, as shown in (a) in FIG. 19 , the mobile phone 1 can continue to display the Take a picture. At this time, the mobile phone 1 may not display any captured images of the slave device in the window 1202 . Currently, the mobile phone 1 can also display the shooting image of the mobile phone 1 in full screen. Alternatively, as shown in (b) of FIG. 19 , while the mobile phone 1 continues to display the shooting screen of the mobile phone 1 in the window 1201 , the mobile phone 1 can also display the shooting screen of a certain slave device in the window 1202 . For example, the mobile phone 1 may continue to display the shooting screen 2 of the television 1 in the window 1202 before the user inputs the audio signal to the mobile phone 1 .

In addition, if the mobile phone 1 determines that the user is currently inputting an audio signal to the mobile phone 1, the mobile phone 1 can also add a sounding mark to the shooting screen of the mobile phone 1 according to the above method. Furthermore, the mobile phone 1 can send the captured image carrying the sounding mark to the slave devices of the mobile phone 1 , that is, the TV 1 and the mobile phone 2 . In this way, the TV 1 and the mobile phone 2 can also display the shooting screen of the mobile phone 1 on their own display interface according to the above method. That is to say, in a distributed shooting scenario, when a user inputs an audio signal to any of the master device and the slave device, the device can notify other devices to display its own shooting screen by adding a sound mark to the shooting screen. , so that both the users of the master device and the slave device can focus on the shooting screen of the slave device with the user's voice input, thereby improving the user experience.

It can be understood that, in the above embodiment, the mobile phone 1 is used as the master device in the distributed shooting scene, and the TV 1 and the mobile phone 2 are the slave devices of the mobile phone 1. It can be understood that the master device in the distributed shooting scene The device and the slave device may be any electronic device having the above-mentioned shooting function, which is not limited in this embodiment of the present application.

It should be noted that, in the above-mentioned embodiments, the Android system is used as an example to illustrate the specific method for realizing the distributed shooting function among the various functional modules. It can be understood that it can also be set in other operating systems (such as Hongmeng system, etc.). The corresponding functional modules implement the above method. As long as the functions implemented by various devices and functional modules are similar to the embodiments of the present application, they fall within the scope of the claims of the present application and their technical equivalents.

As shown in FIG. 20 , an embodiment of the present application discloses an electronic device, and the electronic device may be the above-mentioned main device (eg, a mobile phone). The electronic device may specifically include: a touch screen 2001, the touch screen 2001 includes a touch sensor 2006 and a display screen 2007; one or more processors 2002; a memory 2003; a communication module 2008; one or more cameras 2009; one or more applications programs (not shown); and one or more computer programs 2004, the various devices may be connected by one or more communication buses 2005. Wherein, the above-mentioned one or more computer programs 2004 are stored in the above-mentioned memory 2003 and configured to be executed by the one or more processors 2002, and the one or more computer programs 2004 include instructions, which can be used to execute the above-mentioned Relevant steps performed by the main device (eg, mobile phone 1 ) in the embodiment.

As shown in FIG. 21 , an embodiment of the present application discloses an electronic device, and the electronic device may be the above-mentioned slave device (eg, a speaker). The electronic device may specifically include: a touch screen 2101, the touch screen 2101 includes a touch sensor 2106 and a display screen 2107; one or more processors 2102; a memory 2103; a communication module 2108; one or more cameras 2109; one or more applications programs (not shown); and one or more computer programs 2104, the various devices may be connected by one or more communication buses 2105. Wherein, the above-mentioned one or more computer programs 2104 are stored in the above-mentioned memory 2103 and configured to be executed by the one or more processors 2102, and the one or more computer programs 2104 include instructions, which can be used to execute the above-mentioned Relevant steps performed from a device (eg, TV 1 or mobile phone 2 ) in the embodiment.

From the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be allocated as required. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. For the specific working process of the system, apparatus and unit described above, reference may be made to the corresponding process in the foregoing method embodiments, and details are not described herein again.

Each functional unit in each of the embodiments of the embodiments of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage The medium includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: flash memory, removable hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program codes.

The above are only specific implementations of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the embodiments of the present application shall be covered by this within the protection scope of the application examples. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims (22)

1. A method of shooting, comprising:

   The first device displays a first photographing image on the display interface, where the first photographing image is a photographing image collected by the first device;

   After the first device establishes a network connection with the N electronic devices, the first device instructs the N electronic devices to start capturing and shooting images, where N is an integer greater than 1;

   obtaining, by the first device, a second photographed image collected by a second device, where the second device is one of the N electronic devices;

   If the second shooting picture includes a preset sounding mark, the first device displays the second shooting picture in the display interface, and the sounding mark is used to indicate that the user is speaking.
2. The method according to claim 1, wherein a preset button is provided in the display interface, and the preset button is used to synchronously display the shooting images of multiple devices;

   Before the first device acquires the second shot captured by the second device, the method further includes:

   In response to an operation of the user selecting the preset button, the first device creates a first window and a second window in the display interface, where the first window is used to display the first shot captured by the first device screen, the second window is used to display the shooting screen collected by other devices;

   Wherein, the first device displays the second shooting picture in the display interface, including:

   The first device displays the second shooting picture in the second window, and displays the first shooting picture in the first window.
3. The method according to claim 2, characterized in that, in response to an operation of the user selecting the preset button, further comprising:

   establishing a network connection between the first device and the N electronic devices;

   If the first device and the third device first establish a network connection, before the first device acquires the second shot captured by the second device, the method further includes:

   acquiring, by the first device, a third photographed image collected by the third device, where the third device is an electronic device other than the second device among the N electronic devices;

   The first device displays the third shooting picture in the second window, and displays the first shooting picture in the first window.
4. The method according to claim 3, wherein if the second shooting picture includes a preset sounding mark, the first device displays the second shooting picture on the display interface, comprising:

   If the second shooting picture contains a preset sounding mark, the first device determines whether the time until the third shooting picture is displayed in the second window exceeds a preset time;

   If the preset time is exceeded, the first device displays the second shooting picture in the second window.
5. The method according to claim 3, wherein after the first device displays the second shooting picture in the second window, the method further comprises:

   acquiring, by the first device, a third photographed image collected by the third device;

   If the third shooting picture does not include the sounding identification, the first device continues to display the second shooting picture in the second window, and does not display the third shooting picture.
6. The method according to claim 5, characterized in that, after the first device acquires the third shooting picture collected by the third device, further comprising:

   If the third shooting picture includes the sounding logo, the first device displays the third shooting picture in the second window; or,

   If the third shooting picture includes the sounding identification, the first device creates a third window in the display interface, and displays the third shooting picture in the third window.
7. The method according to any one of claims 2-6, wherein the second window includes a prompt message, and the prompt message is used to prompt that a user corresponding to the second window is making a sound.
8. The method according to claim 1, wherein if the second shooting picture contains a preset sounding mark, the first device displays the second shooting picture on the display interface, comprising:

   If the second shooting picture includes a preset sounding mark, the first device switches the first shooting picture to the second shooting picture in the display interface.
9. The method according to any one of claims 1-8, wherein after the first device establishes a network connection with the N electronic devices, the method further comprises:

   The first device detects whether a user inputs an audio signal to the first device;

   If a user inputs an audio signal to the first device, the first device adds the sounding identifier to the first captured image;

   The first device sends the first shooting picture carrying the sounding identification to the N electronic devices, so that the N electronic devices display the first shooting picture according to the sounding identification.
10. The method according to claim 9, wherein the detecting, by the first device, whether a user inputs an audio signal to the first device, comprises:

    When the first device uses the camera to capture the first shot picture, it uses the microphone to detect the audio signal;

    When the first device detects the audio signal, acquire M first shooting pictures corresponding to the detected audio signal, where M is an integer greater than 1;

    If the mouth shape feature in the M first shooting pictures matches the detected audio signal, the first device determines that a user inputs an audio signal to the first device.
11. The method according to claim 10, wherein when the first device detects an audio signal, acquiring M first shooting pictures corresponding to the detected audio signal, comprising:

    When the first device detects that the loudness of the audio signal is greater than the preset value, acquire M first shooting pictures corresponding to the detected audio signal.
12. The method according to any one of claims 9-11, wherein after the first device detects whether a user inputs an audio signal to the first device, the method further comprises:

    If a user inputs an audio signal to the first device, the first device stops displaying the second shooting picture.
13. A method of shooting, comprising:

    The second device establishes a network connection with the first device;

    The second device uses a camera to capture and capture images, and uses a microphone to detect audio signals;

    When the second device detects that a user inputs an audio signal to the second device, the second device adds a preset sounding mark to the captured image, where the sounding mark is used to indicate that the user is sounding;

    The second device sends the shooting picture carrying the sounding identification to the first device.
14. The method according to claim 13, characterized in that after the second device uses a camera to capture a captured picture and uses a microphone to detect the audio signal, further comprising:

    When the second device detects the audio signal, acquire M shooting pictures corresponding to the detected audio signal, where M is an integer greater than 1;

    If the mouth shape feature in the M shooting pictures matches the detected audio signal, the second device determines that a user inputs an audio signal to the second device.
15. The method according to claim 14, wherein when the second device detects an audio signal, acquiring M shooting pictures corresponding to the detected audio signal, comprising:

    When the second device detects that the loudness of the audio signal is greater than the preset value, acquire M shooting pictures corresponding to the detected audio signal.
16. The method according to any one of claims 13-15, wherein after the second device establishes a network connection with the first device, the method further comprises:

    acquiring, by the second device, a picture taken by the first device;

    If the photographing picture of the first device includes the sounding identifier, the second device displays the photographing picture of the first device on the display interface.
17. The method according to claim 16, wherein the display interface comprises a first window and a second window, wherein the second device displays the shooting picture of the first device in the display interface, comprising:

    The second device displays the shooting picture of the first device in the first window, and displays the shooting picture of the second device in the second window.
18. An electronic device, characterized in that the electronic device is a first device, and the first device includes:

    display screen;

    one or more cameras;

    one or more processors;

    memory;

    communication module;

    Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions, when the instructions are executed by the electronic device, the electronic device is made to perform the execution of claims 1-12. A shooting method performed by the first device in any one of the above.
19. An electronic device, characterized in that the electronic device is a second device, and the second device comprises:

    display screen;

    one or more cameras;

    one or more processors;

    memory;

    communication module;

    Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions, when the instructions are executed by the electronic device, the electronic device is made to perform the execution of claims 13-17 A shooting method performed by the second device in any one of the above.
20. A distributed shooting system, characterized in that the system comprises the electronic device as claimed in claim 18 and the electronic device as claimed in claim 19 .
21. A computer-readable storage medium storing instructions in the computer-readable storage medium, characterized in that, when the instructions are executed on an electronic device, the electronic device is made to execute the method according to claim 1-12 or 13- A photographing method according to any one of 17.
22. A computer program product comprising instructions, characterized in that, when the computer program product is run on an electronic device, the electronic device is caused to perform a process according to any one of claims 1-12 or 13-17. a shooting method.

Images