WO2022227752A1 - 拍照方法及装置 - Google Patents

拍照方法及装置 Download PDF

Info

Publication number
WO2022227752A1
WO2022227752A1 PCT/CN2022/074348 CN2022074348W WO2022227752A1 WO 2022227752 A1 WO2022227752 A1 WO 2022227752A1 CN 2022074348 W CN2022074348 W CN 2022074348W WO 2022227752 A1 WO2022227752 A1 WO 2022227752A1
Authority
WO
WIPO (PCT)
Prior art keywords
photo
cloud
composition
terminal device
edited
Prior art date
Application number
PCT/CN2022/074348
Other languages
English (en)
French (fr)
Inventor
吴进福
陈刚
王妙锋
王硕强
杨坤
Original Assignee
荣耀终端有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 荣耀终端有限公司 filed Critical 荣耀终端有限公司
Publication of WO2022227752A1 publication Critical patent/WO2022227752A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Definitions

  • the embodiments of the present application relate to the field of photographing, and in particular, to a photographing method and device.
  • the camera's shooting interface can display reference lines, such as the grid lines of the nine-square grid.
  • the user can adjust the shooting position and angle of the mobile phone by referring to the nine-square grid line displayed in the shooting interface, so as to complete the composition in different ways such as the rule of thirds composition, diagonal composition, and symmetrical composition. .
  • the above-mentioned method of assisting the user to complete the composition by displaying the reference line in the shooting interface, the guiding role of the reference line is limited, and the final composition is strongly related to the user's shooting experience and techniques, and some users who do not have rich shooting experience may not understand How to adjust the shooting position and angle of the mobile phone according to the reference line to complete the composition in different ways.
  • Embodiments of the present application provide a photographing method and apparatus, which can display composition guide lines in a photographing interface provided by the terminal device in combination with the current photographing scene when a user uses a terminal device to photograph, so as to guide the user in composition.
  • an embodiment of the present application provides a photographing method, and the method is applied to a device-cloud collaboration system.
  • the terminal-cloud collaboration system includes terminal devices and the cloud.
  • the terminal device is connected to the cloud through a wireless network.
  • the method includes:
  • the terminal device acquires and displays a preview image corresponding to the scene to be shot in response to the user's operation; the terminal device collects scene information of the scene to be shot.
  • the terminal device sends the obtained preview image and scene information to the cloud.
  • the cloud According to the preview image and scene information, the cloud generates a composition guide line corresponding to the composition mode suitable for the scene to be shot, and sends the composition guide line to the terminal device.
  • the terminal device displays the composition guide line and the preview image on the shooting interface.
  • the terminal device obtains a photograph corresponding to the scene to be photographed.
  • composition auxiliary line may also be called a reference line, a composition reference line, a composition line, etc., and the name of the composition auxiliary line is not limited herein.
  • the photographing method can display composition auxiliary lines in the photographing interface provided by the terminal device in combination with the current photographing scene when the user uses the terminal device to photograph, so as to guide the user to compose the photograph.
  • the composition guideline can indicate to the user a composition method suitable for the scene to be shot, and guide the user to adjust the shooting position, angle, etc. of the terminal device according to the composition method indicated by the composition guideline to complete the composition.
  • the user can also complete a better composition when taking a photo, and the user experience can be better.
  • the photographing method enables the user to more easily know which composition method to use to compose a picture according to the composition guideline recommended by the cloud when taking a picture with a terminal device, and does not need to judge the composition method by himself, and does not require complicated composition. operate.
  • the scene information can assist the cloud to quickly identify the elements contained in the preview image, improve the efficiency of the cloud generation of composition guide lines, and reduce delays. Time.
  • the sensors include at least a position sensor, an air pressure sensor, a temperature sensor, and an ambient light sensor.
  • the scene information at least includes location information, air pressure information, temperature information, and light intensity information corresponding to the scene to be shot.
  • the terminal device may acquire a preview image corresponding to the scene to be shot according to the first frame rate, and display it on the shooting interface. At the same time, the terminal device collects scene information of the scene to be shot through the sensor according to the first frequency.
  • the terminal device may send the acquired preview image and scene information to the cloud according to the second frequency.
  • the value (or called size) of the second frequency is less than or equal to the minimum value of the value of the first frame rate and the value of the first frequency.
  • the magnitudes of the first frame rate and the first frequency may be the same or different.
  • the frequency at which the terminal device sends scene information to the cloud is less than or equal to the frequency at which the terminal device sends the preview image to the cloud.
  • the terminal device sends the obtained preview image to the cloud according to a third frequency, and sends the obtained scene information to the cloud according to a fourth frequency, where the third frequency is greater than the fourth frequency.
  • the cloud generates, according to the preview image and scene information, a composition guide line corresponding to a composition mode suitable for the scene to be shot, including: the cloud identifies the elements contained in the preview image according to the preview image and the scene information, and records them. The position and proportion of different elements in the preview image. According to the elements contained in the preview image, as well as the positions and proportions of different elements, the cloud determines the composition guide line corresponding to the composition method matching the scene to be shot according to the preset matching rules.
  • the matching rules include the correspondence between at least one type of scene to be photographed and the composition auxiliary line, the elements contained in different types of scenes to be photographed, and the positions and proportions of different elements are different.
  • the above-mentioned matching rules may be artificially defined rules.
  • the elements included in the preview image may be sky, sea water, grass, people, and the like.
  • the position of the element refers to the pixel coordinates of the area where the element is located in the preview image
  • the proportion of the element refers to the ratio of the number of pixels in the area where the element is located in the preview image to the number of pixels in the entire preview image.
  • the cloud identifies the elements contained in the preview image according to the preview image and the scene information, including: the cloud uses the first method to segment the preview image, and then identifies the elements contained in the preview image based on the segmentation result and the scene information. .
  • the scene information is used to assist the cloud to quickly identify the elements contained in the preview image based on the segmentation result.
  • the location information included in the scene information is seaside, it may indicate that the preview image may contain seawater, which can assist the cloud to quickly identify whether the preview image contains seawater based on the segmentation result.
  • the first method may be an edge detection-based method, a wavelet transform-based method, a deep learning-based method, or the like.
  • the first method can try to use traditional segmentation methods or methods with smaller models.
  • the cloud can use the deep learning segmentation network U-NET to segment preview images.
  • the cloud generates, according to the preview image and scene information, a composition guide line corresponding to a composition mode suitable for the scene to be shot, including: the cloud performs saliency detection on the preview image, and extracts the saliency result of the preview image; The cloud inputs the saliency results and scene information of the preview image into the trained artificial intelligence AI network, and obtains the probability distribution of various composition modes corresponding to the preview image output by the AI network; The probability distribution of the mode is determined, and the composition auxiliary line corresponding to the composition mode matching the scene to be shot is determined.
  • the probability of a composition method that is not suitable for the type of scene to be shot is 0 or close to 0.
  • the composition auxiliary line returned by the cloud to the terminal device may be a coordinate data set composed of coordinates of multiple pixels in the preview image.
  • the cloud may also return an image containing the composition guideline to the terminal device.
  • the pixel value of the pixel in the area where the composition guideline is located may be 0, and the pixel value of the area other than the composition guideline may be 0.
  • the pixel value of the pixel point may be P, where P is an integer greater than 0 and less than or equal to 255.
  • the pixel values of the pixel points of the regions other than the composition auxiliary lines may be all 255.
  • the method before the cloud generates a composition guide line corresponding to a composition mode suitable for the scene to be shot according to the preview image and scene information, the method further includes: correcting the preview image by the cloud to obtain a corrected preview image.
  • the cloud generates, according to the preview image and scene information, a composition guide line corresponding to a composition mode suitable for the scene to be shot, including: the cloud generates a composition guide line corresponding to the composition mode suitable for the scene to be shot according to the corrected preview image and scene information.
  • the cloud includes a line detection network and an image correction module; the cloud corrects the preview image to obtain a corrected preview image, including: the cloud inputs the preview image into the line detection network, and detects through the line detection network that the preview image contains The cloud determines the transformation matrix required to correct the preview image according to the lines contained in the preview image through the image correction module, and uses the transformation matrix to correct the preview image to obtain the corrected preview image.
  • the line detection network includes: a backbone network, a connection point prediction module, a line segment sampling module, and a line segment correction module; the line detection network detects the lines included in the preview image, including: after the preview image is input into the backbone network, the backbone The network extracts the features in the preview image, and outputs the shared convolution feature map corresponding to the preview image to the connection point prediction module; the connection point prediction module outputs the candidate connection points corresponding to the preview image according to the shared convolution feature map, and transmits it to the line segment sampling module; The line segment sampling module predicts the lines contained in the preview image according to the candidate connection points.
  • the transformation matrix includes at least a rotation matrix and a homography matrix.
  • the cloud corrects the preview image, so that the cloud can more accurately identify the elements contained in the preview image and the different The position and proportion of the elements, so that the generated auxiliary lines are more in line with the composition method corresponding to the scene to be shot.
  • the method further includes: in response to the user's operation of turning off the display function of the composition auxiliary line, the terminal device does not display the composition auxiliary line.
  • the terminal device sending the acquired preview image to the cloud includes: the terminal device extracts image features of the acquired preview image through a preset convolutional neural network; the terminal device sends the acquired preview image to the cloud. image features.
  • the terminal device Compared with the method of directly sending the preview image to the cloud, the terminal device only sends the image features of the preview image to the cloud, which is used for the cloud to generate the composition guide line corresponding to the composition method suitable for the scene to be shot, which can better protect the user's privacy. Prevent user privacy leakage.
  • the method further includes: the cloud obtains the contour lines of the elements contained in the preview image;
  • the auxiliary line and the outline of the elements contained in the preview image generate a shooting outline suitable for the scene to be shot, and send the shooting outline to the terminal device; after the terminal device receives the shooting outline, it will shoot the outline and composition assistance.
  • the line is displayed on the shooting interface along with the preview image being displayed.
  • the shooting contour line returned by the cloud to the terminal device may also be a coordinate data set composed of multiple pixel coordinates in the preview image (different from the coordinate data set of the composition auxiliary line).
  • the cloud can also return to the terminal device an image containing the composition guideline and the shooting contour line.
  • the pixel value of the pixel in the area where the composition guideline is located may be 0;
  • the pixel value can also be 0.
  • the pixel values of the pixel points of the regions other than the composition auxiliary line and the shooting contour line may all be P, and P is an integer greater than 0 and less than or equal to 255.
  • the pixel values of the pixel points in the area where the composition auxiliary line is located may be different from the pixel values of the pixel points in the area where the photographing contour line is located.
  • the user Before using the terminal device to take a photo, the user can refer to the composition guide displayed on the shooting interface to adjust the shooting position, angle, etc. of the terminal device, and can further refer to the shooting outline displayed on the shooting interface to move the elements in the preview screen. to the position indicated by the shooting outline. In this way, the user's operation of composing pictures can be further simplified, and the user experience can be improved.
  • the method further includes: the terminal device sends a photo corresponding to the scene to be shot to the cloud; the cloud optimizes the composition of the photo, obtains at least one composition-optimized photo corresponding to the photo, and converts the photo to the cloud.
  • the first photo includes the photo and one or more of the at least one composition-optimized photo.
  • the cloud further optimizes the composition of the photo (which can be called the initial photo) taken by the terminal device, and returns the optimized photo to the terminal device, which can improve the success rate of composition, and the photo after composition optimization will not appear rotation, perspective, etc. Distortion, can provide users with better photo choices.
  • the terminal device simultaneously displays the initial photo taken by the user and the photo with optimized composition, which can take into account the needs of the user, so that the user can choose either the photo taken by himself or the photo with optimized composition recommended by the cloud.
  • the cloud includes a line detection network, an image correction module, a saliency detection module, and an aesthetic scoring network.
  • the cloud performs composition optimization on the photo, and obtains at least one composition-optimized photo corresponding to the photo, including: the cloud inputs the photo into a line detection network, and detects lines included in the photo through the line detection network
  • the cloud determines the transformation matrix required to correct the photo through the image correction module according to the lines contained in the photo, and uses the transformation matrix to correct the photo to obtain the corrected photo; the cloud passes the saliency detection
  • the module performs saliency detection on the corrected photo, and obtains the saliency result of the corrected photo; the cloud inputs the corrected photo and the saliency result into the aesthetic scoring network, and obtains the output of the aesthetic scoring network.
  • the terminal device may first detect whether the initial photo is a portrait photo, and if the initial photo is a portrait photo, the terminal device may directly save the portrait photo without sending it to the cloud. Initial photos to better protect user privacy and prevent user privacy leakage.
  • the method further includes: the terminal device sends the first photo to the cloud; and the cloud acquires the user's aesthetic preference according to the first photo.
  • sending the first photo to the cloud by the terminal device includes: the terminal device extracts image features of the first photo through a preset convolutional neural network; the terminal device sends the image features of the first photo to the cloud .
  • sending the first photo by the terminal device to the cloud includes: the terminal device sending the identification information of the first photo to the cloud.
  • the cloud acquiring the user's aesthetic preference according to the first photo includes: the cloud acquiring the first photo stored in the cloud according to the identification information of the first photo, and acquiring the user's aesthetic preference according to the first photo.
  • the cloud obtains the user's aesthetic preference according to the first photo, including: the cloud obtains corresponding scene information according to the first photo, and obtains the user's aesthetic preference according to the first photo and the scene information.
  • the terminal device uploads the user's photo composition selection results to the cloud, and the cloud performs migration training on the composition optimization network to obtain the user's privately customized composition optimization network.
  • This method of private customization of photo composition based on end-cloud collaboration can rely on the strong computing power of the cloud to effectively reduce the hardware requirements of the terminal device, continue to recommend photo composition that is more in line with user habits, reduce user learning costs, and improve user photo experience.
  • an embodiment of the present application provides a terminal-cloud collaboration system, including a terminal device and a cloud; the terminal device is connected to the cloud through a wireless network.
  • the terminal device cooperates with the cloud to implement the method described in the first aspect and any one of the implementation manners of the first aspect.
  • an embodiment of the present application provides a photographing method, and the method is applied to a terminal device, and the terminal device is connected to the cloud through a wireless network.
  • the method includes:
  • the terminal device acquires and displays a preview image corresponding to the scene to be shot in response to the user's operation; at the same time, the terminal device collects scene information of the scene to be shot.
  • the terminal device sends the obtained preview image and scene information to the cloud.
  • the terminal device receives the composition auxiliary line from the cloud, and the composition auxiliary line is used to indicate a composition method suitable for the scene to be shot.
  • the terminal device displays the composition auxiliary line and the preview image being displayed on the shooting interface. In response to the user's photographing operation, the terminal device obtains a photograph corresponding to the scene to be photographed.
  • the camera application program of the terminal device can be started first.
  • the user can click or touch the icon of the camera on the terminal device, and the terminal device can start the camera in response to the user's click or touch operation on the icon of the camera (or the user can also activate the camera through a voice assistant, without limitation).
  • the photographing interface provided by the terminal device through the photographing application program may further include a photographing button, and the essence of the photographing button may be a function control displayed in the photographing interface.
  • the user may click or touch the photographing button, and the terminal device may take a photograph in response to the user's clicking or touching operation of the photographing button, thereby obtaining a photograph corresponding to the scene to be photographed.
  • the sensors include at least a position sensor, an air pressure sensor, a temperature sensor, and an ambient light sensor.
  • the scene information at least includes location information, air pressure information, temperature information, and light intensity information corresponding to the scene to be shot.
  • the terminal device may acquire a preview image corresponding to the scene to be shot according to the first frame rate, and display it on the shooting interface. At the same time, the terminal device collects scene information of the scene to be shot through the sensor according to the first frequency.
  • the terminal device may send the acquired preview image and scene information to the cloud according to the second frequency.
  • the value (or called size) of the second frequency is less than or equal to the minimum value of the value of the first frame rate and the value of the first frequency.
  • the magnitudes of the first frame rate and the first frequency may be the same or different.
  • the frequency at which the terminal device sends scene information to the cloud is less than or equal to the frequency at which the terminal device sends the preview image to the cloud.
  • the terminal device sends the obtained preview image to the cloud at a third frequency (30 times/second), and sends the obtained scene information to the cloud at a fourth frequency (10 times/second), where the third frequency is greater than the fourth frequency.
  • the terminal device displays a composition auxiliary line in the shooting interface, which can guide the user in composition.
  • the composition guide line can indicate to the user a composition method suitable for the scene to be shot, and guide the user to adjust the shooting position, angle, etc.
  • composition guide line to complete the composition, so that there is no rich shooting experience.
  • the users of 100% can also complete a better composition when taking pictures, and the user experience can be better.
  • Based on the composition guides users can more easily know which composition method to use to compose a picture, and they do not need to judge the composition method by themselves, and do not need complex composition operations.
  • the terminal device can display the composition guide line together with the preview image being displayed on the shooting interface, and may also display a text prompt in the shooting interface, prompting the user to complete the composition according to the guide of the composition guide line.
  • the text prompt may also be generated by the cloud when the composition auxiliary line is generated.
  • the cloud can send the composition auxiliary lines and text prompts to the terminal device together.
  • the text prompts may be different.
  • the composition auxiliary line may not be displayed.
  • the terminal device can display the ordinary nine-square grid auxiliary line.
  • the method further includes: in response to the user's operation of turning off the display function of the composition auxiliary line, the terminal device does not display the composition auxiliary line.
  • the terminal device may provide the user with a function control or a physical button for disabling the display function of the composition auxiliary line. operate.
  • the terminal device may not display the composition auxiliary line in response to the operation of the user clicking or touching the aforementioned functional controls or physical keys.
  • the terminal device does not display the composition auxiliary line, which can further consider the needs of more users and improve the user experience.
  • the terminal device may send the preview image to the cloud only when the function of displaying the composition guideline is enabled to obtain the composition guideline returned by the cloud according to the preview image.
  • the composition guideline After the display of the composition guideline is turned off, every time the terminal device takes a photo, the composition guideline will not be displayed in the shooting interface. The terminal device will not display the composition guideline in the shooting interface until the user turns on the display of the composition guideline again.
  • the terminal device sending the acquired preview image to the cloud includes: the terminal device extracts image features of the acquired preview image through a preset convolutional neural network; the terminal device sends the acquired preview image to the cloud. image features.
  • the terminal device Compared with the method of directly sending the preview image to the cloud, the terminal device only sends the image features of the preview image to the cloud, which is used for the cloud to generate the composition guide line corresponding to the composition method suitable for the scene to be shot, which can better protect the user's privacy. Prevent user privacy leakage.
  • the method further includes: the terminal device receives the shooting contour line from the cloud; and the terminal device displays the shooting contour line and the composition auxiliary line together with the preview image being displayed on the shooting interface.
  • the user Before using the terminal device to take a photo, the user can refer to the composition guide displayed on the shooting interface to adjust the shooting position, angle, etc. of the terminal device, and can further refer to the shooting outline displayed on the shooting interface to move the elements in the preview screen. to the position indicated by the shooting outline. In this way, the user's operation of composing pictures can be further simplified, and the user experience can be improved.
  • the method further includes: the terminal device sends a photo corresponding to the scene to be shot to the cloud; the terminal device receives at least one composition-optimized photo corresponding to the photo from the cloud; the terminal device displays the photo, and The at least one composition-optimized photo; the terminal device saves the first photo in response to the user's operation of saving the first photo, where the first photo includes the photo and the at least one composition-optimized photo. one or more.
  • the photo with optimized composition displayed by the terminal device has a higher composition success rate.
  • the photos after the composition optimization will not have distortions such as rotation and perspective, which can provide users with better photo choices.
  • the terminal device simultaneously displays the initial photo taken by the user and the photo with optimized composition, which can take into account the needs of the user, so that the user can choose either the photo taken by himself or the photo with optimized composition recommended by the cloud.
  • the terminal device may first detect whether the initial photo is a portrait photo, and if the initial photo is a portrait photo, the terminal device may directly save the portrait photo without sending it to the cloud. Initial photos to better protect user privacy and prevent user privacy leakage.
  • an embodiment of the present application provides a terminal device, where the terminal device may include a photographing apparatus, and the apparatus may be used to implement the method described in the third aspect.
  • the functions of the apparatus may be implemented by hardware, or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions, for example, a camera module, a display module, a sensor module, a sending module, a receiving module, and the like.
  • the camera module is used to obtain a preview image corresponding to the scene to be shot after starting and running the photographing application; the display module is used to display the preview image obtained by the camera module on the shooting interface; the sensor module is used to collect the to-be-photographed scene.
  • the scene information of the shooting scene is used for sending the preview image obtained by the camera module and the scene information obtained by the sensor module to the cloud; the receiving module is used for receiving the composition auxiliary line from the cloud,
  • the composition guide line is used to indicate a composition mode suitable for the scene to be shot;
  • the display module is further used for displaying the composition guide line and the preview image being displayed on the shooting interface together;
  • the camera module and is further configured to obtain a photo corresponding to the scene to be captured in response to the user's photo-taking operation after receiving the user's photo-taking operation.
  • the frequency of sending the scene information to the cloud by the sending module is less than the frequency of sending the preview image to the cloud.
  • the display module is further configured to not display the composition auxiliary line in response to the user's operation of turning off the display function of the composition auxiliary line.
  • the sending module is specifically configured to extract image features of the obtained preview image through a preset convolutional neural network; and send the obtained image features of the preview image to the cloud.
  • the function of extracting the image features of the obtained preview image through a preset convolutional neural network can also be performed by a separate feature extraction module, for example, the device further includes a feature extraction module.
  • the receiving module is further configured to receive the shooting contour line from the cloud; the display module is further configured to display the shooting contour line and the composition auxiliary line together with the preview image being displayed on the shooting interface.
  • the sending module is further configured to send a photo corresponding to the scene to be shot to the cloud;
  • the receiving module is further configured to receive at least one composition-optimized photo corresponding to the photo from the cloud;
  • the display module is also configured to Displaying the photo and the at least one composition-optimized photo, and in response to the user's operation of saving the first photo, saving the first photo, where the first photo includes the photo and the at least one composition-optimized photo one or more of the following photos.
  • an embodiment of the present application provides an electronic device, including: a processor, a memory for storing instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device The method as described in the third aspect and any one of the implementation manners of the third aspect is implemented.
  • the electronic device can be a mobile terminal, such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an AR/VR device, a notebook computer, a super mobile personal computer, a netbook, a personal digital assistant, etc., or a digital camera, a single-lens reflex camera/ Professional shooting equipment such as mirrorless cameras, action cameras, PTZ cameras, and drones.
  • a mobile terminal such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an AR/VR device, a notebook computer, a super mobile personal computer, a netbook, a personal digital assistant, etc.
  • a digital camera a single-lens reflex camera/ Professional shooting equipment such as mirrorless cameras, action cameras, PTZ cameras, and drones.
  • an embodiment of the present application provides a computer-readable storage medium on which computer program instructions are stored; when the computer program instructions are executed by an electronic device, the electronic device is made to implement any of the third aspect and the third aspect. A method as described in an implementation.
  • the embodiments of the present application further provide a computer program product, including computer-readable codes, when the computer-readable codes are executed in an electronic device, the electronic device can implement the third aspect and the third aspect. Any of the methods described in the implementation.
  • the embodiments of the present application further provide a photographing method, where the method is applied to a cloud, and the cloud is connected to a terminal device through a wireless network; the method includes:
  • the cloud receives the preview image and scene information corresponding to the scene to be shot from the terminal device. According to the preview image and scene information, the cloud generates a composition guide line corresponding to the composition method suitable for the scene to be shot. The cloud sends composition auxiliary lines to the terminal device.
  • composition auxiliary line may also be called a reference line, a composition reference line, a composition line, etc., and the name of the composition auxiliary line is not limited herein.
  • the cloud Based on the preview image and scene information, the cloud generates a composition guide line corresponding to the composition method suitable for the scene to be shot, and sends the composition guide line to the terminal device, so that when the user uses the terminal device to take pictures, the composition guide line can guide the user in composition.
  • the composition guide line which indicates the composition mode suitable for the scene to be shot for the user, and guides the user to adjust the shooting position, angle, etc. of the terminal device according to the composition mode indicated by the composition auxiliary line to complete the composition. Therefore, a user without rich shooting experience can also complete a better composition when taking a photo, and the user experience can be better.
  • the scene information can assist the cloud to quickly identify the elements contained in the preview image, improve the efficiency of the cloud generation of composition guide lines, and reduce delays. Time.
  • the cloud generates, according to the preview image and scene information, a composition guide line corresponding to a composition mode suitable for the scene to be shot, including: the cloud identifies the elements contained in the preview image according to the preview image and the scene information, and records them. The position and proportion of different elements in the preview image. According to the elements contained in the preview image, as well as the positions and proportions of different elements, the cloud determines the composition guide line corresponding to the composition method matching the scene to be shot according to the preset matching rules.
  • the matching rules include the correspondence between at least one type of scene to be photographed and the composition auxiliary line, the elements contained in different types of scenes to be photographed, and the positions and proportions of different elements are different.
  • the above-mentioned matching rules may be artificially defined rules.
  • the elements included in the preview image may be sky, sea water, grass, people, and the like.
  • the position of the element refers to the pixel coordinates of the area where the element is located in the preview image
  • the proportion of the element refers to the ratio of the number of pixels in the area where the element is located in the preview image to the number of pixels in the entire preview image.
  • the cloud identifies the elements contained in the preview image according to the preview image and scene information, including: the cloud uses the first method to segment the preview image, and then identifies the elements contained in the preview image based on the segmentation result and scene information. .
  • the scene information is used to assist the cloud to quickly identify the elements contained in the preview image based on the segmentation result.
  • the location information included in the scene information is seaside, it may indicate that the preview image may contain seawater, which can assist the cloud to quickly identify whether the preview image contains seawater based on the segmentation result.
  • the first method may be an edge detection-based method, a wavelet transform-based method, a deep learning-based method, or the like.
  • the first method can try to use traditional segmentation methods or methods with smaller models.
  • the cloud can use the deep learning segmentation network U-NET to segment preview images.
  • the cloud generates, according to the preview image and scene information, a composition guide line corresponding to a composition mode suitable for the scene to be shot, including: the cloud performs saliency detection on the preview image, and extracts the saliency result of the preview image; The cloud inputs the saliency results and scene information of the preview image into the trained artificial intelligence AI network, and obtains the probability distribution of various composition modes corresponding to the preview image output by the AI network; The probability distribution of the mode is determined, and the composition auxiliary line corresponding to the composition mode matching the scene to be shot is determined.
  • the probability of a composition method that is not suitable for the type of scene to be shot is 0 or close to 0.
  • the method before the cloud generates a composition guide line corresponding to a composition mode suitable for the scene to be shot according to the preview image and scene information, the method further includes: correcting the preview image by the cloud to obtain a corrected preview image.
  • the cloud generates, according to the preview image and scene information, a composition guide line corresponding to a composition mode suitable for the scene to be shot, including: the cloud generates a composition guide line corresponding to the composition mode suitable for the scene to be shot according to the corrected preview image and scene information.
  • the cloud includes a line detection network and an image correction module; the cloud corrects the preview image to obtain a corrected preview image, including: the cloud inputs the preview image into the line detection network, and detects through the line detection network that the preview image contains The cloud determines the transformation matrix required to correct the preview image according to the lines contained in the preview image through the image correction module, and uses the transformation matrix to correct the preview image to obtain the corrected preview image.
  • the line detection network includes: a backbone network, a connection point prediction module, a line segment sampling module, and a line segment correction module; the line detection network detects the lines included in the preview image, including: after the preview image is input into the backbone network, the backbone The network extracts the features in the preview image, and outputs the shared convolution feature map corresponding to the preview image to the connection point prediction module; the connection point prediction module outputs the candidate connection points corresponding to the preview image according to the shared convolution feature map, and transmits it to the line segment sampling module; The line segment sampling module predicts the lines contained in the preview image according to the candidate connection points.
  • the transformation matrix includes at least a rotation matrix and a homography matrix.
  • the cloud corrects the preview image, so that the cloud can more accurately identify the elements contained in the preview image and the different The position and proportion of the elements, so that the generated auxiliary lines are more in line with the composition method corresponding to the scene to be shot.
  • the method further includes: the cloud obtains the contour lines of the elements contained in the preview image; The auxiliary lines and the outlines of the elements contained in the preview image are used to generate shooting outlines suitable for the scene to be shot.
  • the cloud sends the shooting contour line to the terminal device.
  • the shooting contour line sent by the cloud to the terminal device enables the user to refer to the composition guideline displayed in the shooting interface when using the terminal device to take pictures, and adjust the shooting position, angle, etc. of the terminal device, and also refer to the shooting interface.
  • the displayed shooting contour line moves the element in the preview screen to the position indicated by the shooting contour line. This further simplifies the user's operation of composing pictures and improves the user experience.
  • the method further includes: the cloud receiving a photo corresponding to the scene to be shot from the terminal device.
  • the cloud performs composition optimization on the photo, and obtains at least one composition-optimized photo corresponding to the photo.
  • the cloud sends the at least one composition-optimized photo to the terminal device.
  • the cloud further optimizes the composition of the photo (which can be called the initial photo) taken by the terminal device, and returns the optimized photo to the terminal device, which can improve the success rate of composition, and the photo after composition optimization will not appear rotation, perspective, etc. Distortion, can provide users with better photo choices.
  • the composition of the photo which can be called the initial photo
  • the photo after composition optimization will not appear rotation, perspective, etc. Distortion, can provide users with better photo choices.
  • the cloud includes a line detection network, an image correction module, a saliency detection module, and an aesthetic scoring network.
  • the cloud performs composition optimization on the photo, and obtains at least one composition-optimized photo corresponding to the photo, including: the cloud inputs the photo into a line detection network, and detects lines included in the photo through the line detection network
  • the cloud determines the transformation matrix required to correct the photo through the image correction module according to the lines contained in the photo, and uses the transformation matrix to correct the photo to obtain the corrected photo; the cloud passes the saliency detection
  • the module performs saliency detection on the corrected photo, and obtains the saliency result of the corrected photo; the cloud inputs the corrected photo and the saliency result into the aesthetic scoring network, and obtains the output of the aesthetic scoring network.
  • the initial photos from the terminal device received by the cloud are non-portrait photos, such as some landscape/landscape photos.
  • an embodiment of the present application provides a cloud server, and the device can be used to implement the method described in the eighth aspect above.
  • the functions of the apparatus may be implemented by hardware, or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions, for example, a receiving module, a composition module, a sending module, and the like.
  • the receiving module is used to receive the preview image and scene information corresponding to the scene to be shot from the terminal device; the composition module is used to generate the corresponding composition mode suitable for the scene to be shot according to the preview image and the scene information.
  • the composition auxiliary line; the sending module is used for sending the composition auxiliary line to the terminal device.
  • the composition module is specifically configured to identify the elements contained in the preview image according to the preview image and scene information, and record the positions and proportions of different elements in the preview image; according to the elements contained in the preview image, and The positions and proportions of different elements are determined according to preset matching rules, and the composition guide lines corresponding to the composition mode matching the scene to be shot are determined.
  • the matching rules include the correspondence between at least one type of scene to be photographed and the composition auxiliary line, the elements contained in different types of scenes to be photographed, and the positions and proportions of different elements are different.
  • the above-mentioned matching rules may be artificially defined rules.
  • the elements included in the preview image may be sky, sea water, grass, people, and the like.
  • the position of the element refers to the pixel coordinates of the area where the element is located in the preview image
  • the proportion of the element refers to the ratio of the number of pixels in the area where the element is located in the preview image to the number of pixels in the entire preview image.
  • composition module is specifically configured to use the first method to segment the preview image, and then identify the elements contained in the preview image based on the segmentation result and scene information.
  • the scene information is used to assist the composition module to quickly identify the elements contained in the preview image based on the segmentation result.
  • the location information included in the scene information is seaside, it may indicate that the preview image may contain seawater, which can assist the composition module to quickly identify whether the preview image contains seawater based on the segmentation result.
  • the first method may be an edge detection-based method, a wavelet transform-based method, a deep learning-based method, or the like. Considering the high performance requirements and low accuracy requirements, the first method can try to use traditional segmentation methods or methods with smaller models.
  • the composition module can use the deep learning segmentation network U-NET to segment the preview image.
  • the composition module is specifically used to perform saliency detection on the preview image and extract the saliency result of the preview image; input the saliency result and scene information of the preview image into the trained artificial intelligence AI network to obtain AI The probability distribution of the various composition modes corresponding to the preview image output by the network; according to the probability distribution of the various composition modes corresponding to the preview image output by the AI network, determine the composition guide line corresponding to the composition mode that matches the scene to be shot.
  • the probability of a composition method that is not suitable for the type of scene to be shot is 0 or close to 0.
  • the composition module is specifically configured to correct the preview image to obtain a corrected preview image; and according to the corrected preview image and scene information, generate a composition guide line corresponding to a composition mode suitable for the scene to be shot.
  • the cloud includes a line detection network and an image correction module; a composition module is specifically configured to input the preview image into the line detection network, and detect the lines contained in the preview image through the line detection network; line, determine the transformation matrix required to correct the preview image, and use the transformation matrix to correct the preview image to obtain the corrected preview image.
  • the line detection network includes: a backbone network, a connection point prediction module, a line segment sampling module, and a line segment correction module; a composition module, which is specifically used to input the preview image into the backbone network, the backbone network extracts the features in the preview image, and outputs the preview.
  • the shared convolution feature map corresponding to the image is sent to the connection point prediction module; the connection point prediction module outputs the candidate connection points corresponding to the preview image according to the shared convolution feature map, and transmits it to the line segment sampling module; the line segment sampling module predicts the preview according to the candidate connection points Image contains lines.
  • the transformation matrix includes at least a rotation matrix and a homography matrix.
  • the composition module is further configured to obtain the contour lines of the elements contained in the preview image; and generate the shooting contour lines suitable for the scene to be shot according to the composition auxiliary lines and the contour lines of the elements contained in the preview image.
  • the sending module is also used for sending the shooting contour line to the terminal device.
  • the apparatus further includes: a composition optimization module; and a receiving module, further configured to receive a photo corresponding to the scene to be shot from the terminal device.
  • a composition optimization module configured to optimize the composition of the photo to obtain at least one composition-optimized photo corresponding to the photo.
  • the sending module is further configured to send the at least one composition-optimized photo to the terminal device.
  • the cloud includes a line detection network, an image correction module, a saliency detection module, and an aesthetic scoring network.
  • the composition optimization module is specifically configured to input the photo into a line detection network, and detect the lines included in the photo through the line detection network; determine the necessary parameters to correct the photo according to the lines included in the photo through the image correction module.
  • the required transformation matrix is used, and the transformation matrix is used to rectify the photo to obtain the rectified photo; the saliency detection module is used to detect the saliency of the rectified photo, and the saliency result of the rectified photo is obtained.
  • the photo after the described correction and the significant result are input into the aesthetics scoring network, obtain a plurality of candidate cut-outs of the output of the aesthetics-scoring network and the corresponding score of each described candidate cut-out; Determine multiple candidate cuts At least one candidate crop with the highest score in the picture is the photo after composition optimization.
  • the initial photos from the terminal device received by the receiving module are non-portrait photos, such as some landscape/landscape photos.
  • an embodiment of the present application provides an electronic device, including: a processor, a memory for storing instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device The method as described in the eighth aspect and any one of the implementation manners of the eighth aspect is implemented.
  • the electronic device may be a cloud server, a server cluster, a cloud platform, and the like.
  • an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored; when the computer program instructions are executed by an electronic device, the electronic device is made to implement the eighth and eighth aspects. Any of the methods described in the implementation.
  • the embodiments of the present application further provide a computer program product, including computer-readable codes, when the computer-readable codes are executed in an electronic device, the electronic device can implement the eighth aspect and the eighth aspect.
  • a computer program product including computer-readable codes
  • the electronic device can implement the eighth aspect and the eighth aspect. The method described in any one of the implementations of .
  • an embodiment of the present application provides a photographing method, which is applied to a device-cloud collaboration system.
  • the terminal-cloud collaboration system includes terminal equipment and the cloud, and the terminal equipment is connected to the cloud through a wireless network.
  • the method includes:
  • the terminal device acquires the photo to be edited and the scene information corresponding to the photo to be edited; the terminal device sends the photo to be edited and the scene information to the cloud.
  • the cloud performs composition optimization on the photo to be edited according to the photo to be edited and scene information, obtains at least one photo with optimized composition corresponding to the photo to be edited, and sends the at least one photo with optimized composition to the terminal device.
  • the terminal device displays the photo to be edited and the at least one composition-optimized photo. For the above-mentioned photos displayed by the terminal device, the user performs a saving operation on the first photo whose composition is satisfactory.
  • the terminal device selects the first photo in response to the user's operation.
  • the terminal device sends the first photo to the cloud.
  • the cloud obtains the user's aesthetic preference according to the first photo.
  • the first photo includes an initial photo and one or more of the at least one composition-optimized photo.
  • the cloud includes a line detection network, an image correction module, a saliency detection module, and an aesthetic scoring network; the cloud optimizes the composition of the photo to be edited to obtain at least one composition corresponding to the photo to be edited.
  • the optimized photo includes: the cloud inputs the image to be edited into the line detection network, and detects the lines contained in the image to be edited through the line detection network; the cloud uses the image correction module according to the image to be edited. For the lines included in the image, determine the transformation matrix required to correct the preview image, and use the transformation matrix to correct the image to be edited to obtain the corrected image to be edited.
  • the line detection network includes: a backbone network, a connection point prediction module, a line segment sampling module, and a line segment correction module; the line detection network detects the lines included in the image to be edited, including: After the image to be edited is input to the backbone network, the backbone network extracts the features in the image to be edited, and outputs the shared convolution feature map corresponding to the image to be edited to the connection point prediction module; the connection point prediction module according to The shared convolution feature map outputs the candidate connection points corresponding to the image to be edited, and transmits it to the line segment sampling module; the line segment sampling module predicts the lines included in the image to be edited according to the candidate connection points.
  • the terminal device uploads the photo or the existing photo taken by the user to the cloud
  • the cloud returns the composition-optimized photo to the terminal device based on the aesthetic scoring network
  • the terminal device uploads the user's selection result of the photo composition to the cloud
  • the cloud Perform transfer training on the aesthetic scoring network to obtain the user's aesthetic preference.
  • an embodiment of the present application provides a photographing method, where the method is applied to a terminal device, and the terminal device is connected to the cloud through a wireless network.
  • the method includes: a terminal device, in response to a user's operation, acquires a photo to be edited and scene information corresponding to the photo to be edited.
  • the terminal device sends the photo to be edited and scene information to the cloud, and the photo to be edited and the scene information are used by the cloud to optimize the composition of the photo to be edited to obtain at least one photo with optimized composition corresponding to the photo to be edited.
  • the terminal device receives the at least one composition-optimized photo sent by the cloud.
  • the terminal device displays the photo to be edited and the at least one photo whose composition is optimized.
  • the terminal device selects the first photo in response to the user's selection operation, and the first photo includes one or more of the photo to be edited and the at least one composition-optimized photo.
  • the terminal device sends the saved first photo to the cloud, and the first photo is used for the cloud to obtain the user's aesthetic preference.
  • sending the first photo to the cloud by the terminal device includes: the terminal device extracts image features of the first photo through a preset convolutional neural network; the terminal device sends the first photo to the cloud. Describe the image features of the first photo.
  • sending the first photo by the terminal device to the cloud includes: sending, by the terminal device, identification information of the first photo to the cloud.
  • an embodiment of the present application provides a terminal device, where the terminal device may include a photographing apparatus, and the apparatus may be used to implement the method described in the fourteenth aspect above.
  • the functions of the apparatus may be implemented by hardware, or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions, for example, a camera module, a sending module, a receiving module, a display module, a processing module, and the like.
  • the camera module is used for obtaining the photo to be edited and scene information corresponding to the photo to be edited in response to the user's operation.
  • a sending module configured to send the photo to be edited and scene information to the cloud, and the photo to be edited and the scene information are used by the cloud to optimize the composition of the photo to be edited to obtain the corresponding image of the photo to be edited.
  • At least one photo with optimized composition At least one photo with optimized composition.
  • a receiving module configured to receive the at least one composition-optimized photo sent by the cloud.
  • a display module configured to display the photo to be edited and the at least one photo whose composition has been optimized.
  • a processing module configured to select the first photo in response to the user's selection operation on the first photo, where the first photo includes one or more of the photo and the at least one composition-optimized photo open.
  • the sending module is further configured to send the saved first photo to the cloud, where the first photo is used for the cloud to obtain the user's aesthetic preference.
  • an embodiment of the present application provides an electronic device, including: a processor, a memory for storing instructions executable by the processor; the processor is configured to execute the instructions, causing the electronic device
  • the device implements the method as described in the fourteenth aspect and any one of the implementation manners of the fourteenth aspect.
  • the electronic device can be a mobile terminal, such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an AR/VR device, a notebook computer, a super mobile personal computer, a netbook, a personal digital assistant, etc., or a digital camera, a single-lens reflex camera/ Professional shooting equipment such as mirrorless cameras, action cameras, PTZ cameras, and drones.
  • a mobile terminal such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an AR/VR device, a notebook computer, a super mobile personal computer, a netbook, a personal digital assistant, etc.
  • a digital camera a single-lens reflex camera/ Professional shooting equipment such as mirrorless cameras, action cameras, PTZ cameras, and drones.
  • an embodiment of the present application further provides a photographing method, where the method is applied to a cloud, and the cloud is connected to a terminal device through a wireless network.
  • the method includes: receiving, by the cloud, photo to-be-edited and scene information from the terminal device.
  • the cloud performs composition optimization on the photo to be edited according to the photo to be edited and scene information, obtains at least one photo with optimized composition corresponding to the photo to be edited, and converts the at least one photo with optimized composition sent to the terminal device.
  • the cloud receives a first photo from the terminal device, where the first photo includes the photo and one or more of the at least one composition-optimized photo.
  • the cloud acquires the user's aesthetic preference according to the first photo.
  • an embodiment of the present application provides an electronic device, and the apparatus can be used to implement the method described in the seventeenth aspect.
  • the functions of the apparatus may be implemented by hardware, or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions, for example, a receiving module, a processing module, a sending module, and the like.
  • the receiving module is configured to receive the photos to be edited and scene information from the terminal device.
  • the processing module is configured to optimize the composition of the photo to be edited according to the photo to be edited and scene information, and obtain at least one photo with optimized composition corresponding to the photo to be edited.
  • a sending module configured to send the at least one composition-optimized photo to the terminal device.
  • the receiving module is further configured to receive a first photo from the terminal device, where the first photo includes the photo and one or more of the at least one composition-optimized photo.
  • the processing module is further configured for the cloud to acquire the user's aesthetic preference according to the first photo.
  • an embodiment of the present application provides an electronic device, including: a processor, a memory for storing instructions executable by the processor; the processor is configured to execute the instructions, causing the electronic device
  • the device implements the method as described in the seventeenth aspect and any one of the implementation manners of the seventeenth aspect.
  • the electronic device may be a cloud server, a server cluster, a cloud platform, and the like.
  • an embodiment of the present application provides a device-cloud collaboration system, including the terminal device described in any one of the implementation manners of the fifteenth aspect and the fifteenth aspect, and the seventeenth aspect and the tenth aspect.
  • the cloud described in any one of the implementation manners in the seven aspects; or, including the terminal device described in any one of the implementation manners in the sixteenth aspect and the sixteenth aspect and the eighteenth aspect and the tenth aspect The cloud described in any one implementation manner of the eight aspects.
  • an embodiment of the present application provides a computer-readable storage medium on which computer program instructions are stored; when the computer program instructions are executed by an electronic device, the electronic device can realize the fourteenth aspect and the fourteenth aspect.
  • the method described in any one of the implementation manners in the aspect; or, the electronic device is caused to implement the method described in any one of the implementation manners of the seventeenth aspect and the seventeenth aspect.
  • an embodiment of the present application further provides a computer program product, including computer-readable codes, when the computer-readable codes are executed in an electronic device, the electronic device can implement the aforementioned fourteenth and tenth aspects The method described in any one of the four aspects is implemented; or, an electronic device is required to implement the method described in any one of the seventeenth aspect and the seventeenth aspect.
  • an embodiment of the present application provides a photographing method, which is applied to a device-cloud collaboration system.
  • the terminal-cloud collaboration system includes terminal equipment and the cloud, and the terminal equipment is connected to the cloud through a wireless network.
  • the method includes:
  • the terminal device in response to the user's editing operation, edits the photo to be edited, and obtains the edited photo.
  • the terminal device sends the photo to be edited and the edited photo to the cloud.
  • the cloud obtains the user's aesthetic preferences according to the photos to be edited and the edited photos, and optimizes the network to obtain basic image quality optimization parameters according to the user-customized photo style.
  • the cloud sends and sends the basic image quality optimization parameters to the terminal device. After receiving the basic image quality optimization parameters, the terminal device updates the local basic image quality parameters.
  • the terminal device uploads the photos before and after editing by the user to the cloud, trains the photo-style optimization network deployed in the cloud, migrates to obtain the photo-style optimization network customized by the user, and synchronously updates the basic image quality algorithm on the terminal device side.
  • This kind of private customization method of photo-taking style based on end-cloud collaboration can rely on the strong computing power of the cloud to effectively reduce the hardware requirements of terminal equipment, and use the cloud to model and update basic image quality algorithms to continuously recommend photo-taking styles that are more in line with user habits. Improve the user's photo experience.
  • an embodiment of the present application provides a photographing method, where the method is applied to a terminal device, and the terminal device is connected to the cloud through a wireless network. The method includes:
  • the terminal device acquires the initial photo and the edited first photo in response to the user's editing operation.
  • the terminal device sends the initial photo and the first photo to the cloud.
  • the cloud obtains a user-customized photographing style optimization network according to the initial photo and the first photo, and obtains basic image quality optimization parameters according to the user-customized photographing style optimization network.
  • the cloud sends and sends the basic image quality optimization parameter to the terminal device.
  • the terminal device updates local basic image quality parameters according to the basic image quality optimization parameters.
  • an embodiment of the present application provides a terminal device, where the terminal device may include a photographing apparatus, and the apparatus may be used to implement the method described in the twenty-fourth aspect above.
  • the functions of the apparatus may be implemented by hardware, or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions, for example, a camera module, a sending module, a receiving module, a storage module, and the like.
  • the camera module is used for acquiring the initial photo and the edited first photo in response to the user's operation.
  • the sending module is used to send the initial photo and the first photo to the cloud.
  • the receiving module is used to receive basic image quality optimization parameters from the cloud.
  • the storage module is used to update the local basic image quality parameters according to the basic image quality optimization parameters.
  • an embodiment of the present application provides an electronic device, including: a processor, a memory for storing instructions executable by the processor; when the processor is configured to execute the instructions, the processor causes the The electronic device implements the method as described in any one of the twenty-fourth aspect and the implementation manner of the twenty-fourth aspect.
  • the electronic device can be a mobile terminal, such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an AR/VR device, a notebook computer, a super mobile personal computer, a netbook, a personal digital assistant, etc., or a digital camera, a single-lens reflex camera/ Professional shooting equipment such as mirrorless cameras, action cameras, PTZ cameras, and drones.
  • a mobile terminal such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an AR/VR device, a notebook computer, a super mobile personal computer, a netbook, a personal digital assistant, etc.
  • a digital camera a single-lens reflex camera/ Professional shooting equipment such as mirrorless cameras, action cameras, PTZ cameras, and drones.
  • an embodiment of the present application further provides a photographing method, where the method is applied to a cloud, and the cloud is connected to a terminal device through a wireless network; the method includes:
  • the cloud receives the initial photo and the edited first photo from the terminal device.
  • the cloud obtains a user-customized photographing style optimization network according to the initial photo and the first photo, and obtains basic image quality optimization parameters according to the user-customized photographing style optimization network.
  • the cloud sends the basic image quality optimization parameters to the terminal device.
  • an embodiment of the present application provides an electronic device, and the apparatus can be used to implement the method described in the twenty-seventh aspect.
  • the functions of the apparatus may be implemented by hardware, or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions, for example, a receiving module, a processing module, a sending module, and the like.
  • an embodiment of the present application provides an electronic device, including: a processor, a memory for storing instructions executable by the processor; when the processor is configured to execute the instructions, the processor causes the The electronic device implements the method described in any one of the twenty-seventh aspect and the twenty-seventh aspect.
  • the electronic device may be a cloud server, a server cluster, a cloud platform, and the like.
  • an embodiment of the present application provides a device-cloud collaboration system, including the terminal device described in any one of the twenty-fifth aspect and the twenty-fifth aspect, and the twenty-eighth aspect. and the cloud described in any one of the implementations of the twenty-eighth aspect; or, including the terminal device described in any one of the twenty-sixth and twenty-sixth aspects of the implementation and the The cloud described in any one implementation manner of the twenty-ninth aspect and the twenty-ninth aspect.
  • an embodiment of the present application provides a computer-readable storage medium on which computer program instructions are stored; when the computer program instructions are executed by an electronic device, the electronic device is made to implement the methods described in the twenty-fourth aspect and the second The method described in any one of the implementation manners of the fourteenth aspect; or, the electronic device is caused to implement the method described in any one of the implementation manners of the twenty-seventh aspect and the twenty-seventh aspect.
  • an embodiment of the present application further provides a computer program product, including computer-readable codes, which, when the computer-readable codes are executed in an electronic device, enable the electronic device to implement the foregoing twenty-fourth aspect and the first The method described in any one of the implementation manners of the twenty-fourth aspect; or, the electronic device is caused to implement the method described in any one of the implementation manners of the twenty-seventh aspect and the twenty-seventh aspect.
  • FIG. 1 shows a schematic diagram of a mobile phone shooting interface
  • FIG. 2 shows a schematic structural diagram of a device-cloud collaboration system provided by an embodiment of the present application
  • FIG. 2A shows a schematic flowchart of the cloud correcting the preview image in the embodiment of the present application
  • FIG. 2B shows a schematic structural diagram of a line detection network provided by an embodiment of the present application
  • FIG. 2C shows a schematic diagram of the effect of image correction provided by an embodiment of the present application.
  • FIG. 2D shows a schematic diagram of another effect of image correction provided by an embodiment of the present application.
  • FIG. 3 shows a schematic structural diagram of a terminal device provided by an embodiment of the present application
  • FIG. 4 shows a schematic flowchart of a photographing method provided by an embodiment of the present application
  • FIG. 5 shows a schematic diagram of a shooting scene provided by an embodiment of the present application
  • FIG. 6 shows a schematic diagram of a shooting interface provided by an embodiment of the present application.
  • FIG. 7 shows another schematic diagram of a shooting interface provided by an embodiment of the present application.
  • FIG. 8 shows a schematic diagram of a photographing operation provided by an embodiment of the present application.
  • FIG. 9 shows another schematic diagram of a shooting interface provided by an embodiment of the present application.
  • FIG. 10 shows a schematic diagram of a setting interface provided by an embodiment of the present application.
  • FIG. 11 shows another schematic diagram of a shooting interface provided by an embodiment of the present application.
  • FIG. 12 shows another schematic diagram of a shooting interface provided by an embodiment of the present application.
  • FIG. 13 shows another schematic diagram of a shooting interface provided by an embodiment of the present application.
  • FIG. 14 shows another schematic flowchart of the photographing method provided by the embodiment of the present application.
  • FIG. 15 shows a schematic flowchart of composition optimization of an initial photo by the cloud in an embodiment of the present application
  • FIG. 16 shows a schematic diagram of a regression principle provided by an embodiment of the present application.
  • FIG. 17 shows a schematic diagram of composition optimization provided by an embodiment of the present application.
  • FIG. 18 shows a schematic diagram of a photo display interface provided by an embodiment of the present application.
  • FIG. 19 shows a schematic structural diagram of a photographing device provided by an embodiment of the present application.
  • FIG. 20 shows another schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 21 shows another schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 22 shows another schematic flowchart of the photographing method provided by an embodiment of the present application.
  • FIG. 23 shows another schematic flowchart of the photographing method provided by the embodiment of the present application.
  • FIG. 24 shows a schematic diagram of a photo editing interface provided by an embodiment of the present application.
  • FIG. 25 shows another schematic flowchart of the photographing method provided by the embodiment of the present application.
  • FIG. 26A shows another schematic diagram of a photo editing interface provided by an embodiment of the present application.
  • FIG. 26B shows the second schematic diagram of the photo editing interface provided by the embodiment of the present application.
  • FIG. 26C shows another schematic diagram No. 3 of the photo editing interface provided by the embodiment of the present application.
  • FIG. 27A shows another schematic diagram of a photo editing interface provided by an embodiment of the present application.
  • FIG. 27B shows the second schematic diagram of the photo editing interface provided by the embodiment of the present application.
  • FIG. 27C shows the third schematic diagram of the photo editing interface provided by the embodiment of the present application.
  • FIG. 28 shows another schematic structural diagram of the photographing device provided by the embodiment of the present application.
  • FIG. 29 shows another schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 30 shows another schematic structural diagram of the photographing device provided by the embodiment of the present application.
  • FIG. 31 shows another schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • references in this specification to "one embodiment” or “some embodiments” and the like mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application.
  • appearances of the phrases “in one embodiment,” “in some embodiments,” “in other embodiments,” “in other embodiments,” etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean “one or more but not all embodiments” unless specifically emphasized otherwise.
  • the terms “including”, “including”, “having” and their variants mean “including but not limited to” unless specifically emphasized otherwise.
  • the term “connected” includes both direct and indirect connections unless otherwise specified.
  • first and second are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features.
  • a feature defined as “first” or “second” may expressly or implicitly include one or more of that feature.
  • words such as “exemplarily” or “for example” are used to represent examples, illustrations or illustrations. Any embodiment or design described in the embodiments of the present application as “exemplarily” or “such as” should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as “exemplarily” or “such as” is intended to present the related concepts in a specific manner.
  • the mobile phone can collect the preview image corresponding to the scene to be shot, and display the preview image on the camera's shooting interface, so that the user can see the preview screen of the scene to be shot (that is, the preview image is in the screen displayed in the shooting interface).
  • Users can adjust shooting parameters such as sensitivity, aperture, and composition according to the preview screen displayed in the shooting interface.
  • Adjusting the composition mode may refer to adjusting the position and proportion of each element in the preview image (eg, characters, buildings, animals, etc.) in the preview image.
  • the user can click the camera button, and the mobile phone can respond to the user's click operation on the camera button, according to the sensitivity, aperture and other shooting parameters of the preview screen, and Preview the composition mode of the screen, take a picture of the scene to be shot, and obtain a photo corresponding to the scene to be shot.
  • shooting parameters such as sensitivity, aperture, and composition
  • the adjustment of the composition mode will affect the quality of the final captured photo. For example, a good composition can make a better photo, while a wrong composition can result in a photo that is not as good as expected. What's more, a better composition when taking pictures can often turn corruption into magic and make up for the shortcomings of users' lack of experience in taking pictures.
  • the mobile phone can display a preset reference line in the camera's shooting interface when the camera starts to run, so that the user can complete the composition according to the reference line.
  • FIG. 1 shows a schematic diagram of a mobile phone camera interface. As shown in Figure 1, when the mobile phone starts up the camera, the reference line displayed in the camera's shooting interface is the grid line 101 of the nine-square grid.
  • the user can adjust the shooting position and angle of the mobile phone by referring to the grid lines 101 of the nine-square grid displayed in the shooting interface, so as to complete the composition of the rule of thirds, the diagonal composition, the symmetrical composition and other different methods. composition.
  • the grid lines 101 of the nine-square grid shown in FIG. 1 are arranged in a "well" shape, and the preview screen is divided into nine equal parts. Generally, it can be considered as a human visual interest point.
  • the user can adjust the shooting position, angle, etc. of the mobile phone when composing the picture, so that the shooting subject in the preview screen is near the intersection (or, the shooting subject can also be near the two vertical lines), so as to complete the composition of the rule of thirds.
  • the guiding role of the reference line is limited, and the final composition is strongly related to the user's shooting experience and techniques.
  • the shooting scene includes the sky and the sea
  • there is a clear dividing line between the sky and the sea it may be more suitable for symmetrical composition.
  • Users with certain shooting experience can adjust the shooting position and angle of the mobile phone to make the sky and the sea.
  • the dividing line is in the middle of the two horizontal lines in the nine-square grid line to achieve symmetrical composition.
  • users without rich shooting experience they do not know how to combine the shooting scene to complete the composition.
  • the mobile phone assists the user to complete the composition by displaying the preset reference line in the shooting interface, and the final composition effect varies from person to person, and cannot help more users to complete a better composition.
  • the embodiment of the present application provides a photographing method, which can display a composition guide line in the photographing interface provided by the terminal device in combination with the current photographing scene when the user uses a terminal device with a photographing function to take a photograph.
  • the user guides the composition.
  • the composition guideline can indicate the composition method suitable for the scene to be shot for the user, and guide the user to adjust the shooting position, angle, etc. of the mobile phone according to the composition method indicated by the composition guideline to complete the composition, so that users without rich shooting experience can be made. It can also complete a better composition when taking pictures, and the user experience can be better.
  • the method can be applied to a terminal-cloud collaborative system composed of terminal equipment and cloud.
  • the "end” of the end-cloud collaboration refers to the terminal device, and the “cloud” refers to the cloud.
  • the cloud can also be called a cloud server, a remote server or a cloud platform.
  • FIG. 2 shows a schematic structural diagram of a terminal-cloud collaboration system provided by an embodiment of the present application.
  • the terminal-cloud collaboration system may include: a terminal device 210 and a cloud 220 , and the terminal device 210 The network is connected to the cloud 220 .
  • the terminal device 210 has a photographing function.
  • the terminal device 210 may be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR)/virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (ultra- Mobile personal computer, UMPC), netbook, personal digital assistant (PDA) and other mobile terminals, or, can also be professional digital cameras, SLR cameras/mirror cameras, action cameras, PTZ cameras, drones, etc.
  • the specific type of the terminal device 210 is not limited in this embodiment of the present application.
  • the terminal device 210 when the terminal device 210 is a shooting device such as a pan-tilt camera, a drone, etc., the terminal device 210 will also include a display device that can provide a shooting interface for displaying composition auxiliary lines.
  • the display device of the gimbal camera can be a mobile phone
  • the display device of the aerial photography drone can be a remote control device, etc.
  • the cloud 220 may be a computer, a server, or a server cluster composed of multiple servers, etc.
  • the application does not limit the implementation architecture of the cloud 220 .
  • the specific form of the terminal device 210 reference may be made to the description in the foregoing embodiments, and details are not repeated here.
  • the photographing method provided by the embodiment of the present application may be as follows:
  • the terminal device 210 acquires a preview image corresponding to the scene to be photographed according to the first frame rate when taking a picture (for example, after starting the photographing application program), and displays it on the photographing interface. At the same time, the terminal device 210 collects scene information of the scene to be shot through the sensor according to the first frequency. In the foregoing process, the terminal device 210 sends the acquired preview image and scene information to the cloud 220 according to the second frequency. After each time the cloud 220 receives the preview image and scene information, according to the preview image and scene information received this time, a composition guide line corresponding to a composition mode suitable for the scene to be shot is generated, and the composition guide line is sent to the terminal device 210 .
  • the terminal device 210 After receiving the composition guideline, the terminal device 210 displays the composition guideline together with the preview image being displayed on the shooting interface. The user may adjust the shooting position, angle, etc. of the terminal device 210 to complete the composition by referring to the composition guideline displayed in the shooting interface. After completing the composition, the user may perform a photographing operation on the terminal device 210 to trigger the terminal device 210 to take a photograph.
  • the magnitude of the second frequency is less than or equal to the minimum value of the first frame rate and the first frequency.
  • the magnitudes of the first frame rate and the first frequency may be the same or different, and the present application does not limit the magnitudes of the first frame rate and the first frequency.
  • the first frame rate may be 30 frames per second (FPS).
  • the first frequency may be 30 times/second.
  • the first frequency can be 10 times/second, 15 times/second, 35 times/second, etc.
  • the first frequency can be smaller than the first frame rate or greater than the first frame Rate.
  • reference may be made to the first frequency, which will not be illustrated again.
  • the terminal device 210 may be configured with sensors such as a position sensor, an air pressure sensor, a temperature sensor, and an ambient light sensor.
  • the scene information of the scene to be shot collected by the terminal device 210 through the sensor may include: location information, air pressure information, temperature information, light intensity information, etc. corresponding to the scene to be shot.
  • This embodiment of the present application does not limit the number and types of sensors captured by the terminal device 210 and the scene information of the scene to be captured that is collected by the terminal device 210 through the sensors.
  • the cloud 220 For each received preview image and scene information, the cloud 220 generates a composition guide line corresponding to a composition mode suitable for the scene to be shot according to the preview image and scene information as follows:
  • the cloud 220 is preset with matching rules for generating a composition guide line corresponding to a composition mode suitable for the scene to be shot according to the elements contained in the preview image and the positions and proportions of different elements.
  • the matching rules may be artificially defined according to photography rules. For example, which shooting scenes are suitable for which composition methods can be manually defined, and the composition guide lines corresponding to the composition methods can be set for the shooting scenes, and then the aforementioned matching rules can be configured in the cloud.
  • the cloud 220 can first identify the elements contained in the preview image according to the preview image and scene information, and record the positions and proportions of different elements in the preview image. Then, the cloud 220 can determine the composition guide line corresponding to the composition mode suitable for the scene to be shot according to the elements contained in the preview image and the positions and proportions of different elements in the aforementioned preset matching rules.
  • the elements included in the preview image may be sky, sea water, grass, people, and the like.
  • the position of the element refers to the pixel coordinates of the area where the element is located in the preview image
  • the proportion of the element refers to the ratio of the number of pixels in the area where the element is located in the preview image to the number of pixels in the entire preview image.
  • the cloud 220 identifies elements contained in the preview image according to the preview image and scene information, and records the positions and proportions of different elements in the preview image, which may include: the cloud 220 adopts the first step.
  • a method divides the preview image, and then identifies the elements contained in the preview image based on the segmentation result and scene information, and records the positions and proportions of different elements in the preview image.
  • the scene information can be used to assist the cloud 220 to quickly identify the elements contained in the preview image based on the segmentation result. For example, when the location information included in the scene information is seaside, it may indicate that the preview image may contain seawater, which can assist the cloud 220 to quickly identify whether the preview image contains seawater based on the segmentation result. Similarly, air pressure information, light information, etc. can be used to assist the cloud 220 to quickly identify the elements included in the preview image based on the segmentation result, and will not be listed one by one.
  • the above-mentioned first method may be an edge detection-based method, a wavelet transform-based method, a deep learning-based method, or the like.
  • the first method may use a traditional segmentation method or a method with a smaller model as much as possible. segmentation.
  • the present application does not limit the specific type of the first method.
  • the cloud 220 presets a matching rule for generating a composition guide line corresponding to a composition mode suitable for the scene to be shot, which may include the following a) to one or more of e).
  • the cloud 220 can combine one or more of the following a) to e), and determine the composition assistance corresponding to the composition mode suitable for the scene to be shot according to the elements contained in the preview image, as well as the positions and proportions of different elements. Wire.
  • the preview image contains two elements such as sky, sea water, grass, mountains, etc., and there is a clear dividing line between the two elements, such as sea level, skyline, etc., then determine the one suitable for the scene to be shot.
  • the composition method is symmetrical composition, and a composition auxiliary line corresponding to the symmetrical composition method is generated in combination with the scene to be shot.
  • the composition method of the shooting scene is a three-point composition, and a composition auxiliary line corresponding to the three-point composition method is generated in combination with the scene to be shot.
  • the cloud can determine that the composition method suitable for the scene to be shot is the golden section composition or the center composition, combined with the scene to be shot. (eg: target the subject) to generate the composition guide line corresponding to the golden section composition method or the center composition method.
  • the composition method suitable for the scene to be shot is the guide line composition, and combine the scene to be shot (such as: along curved roads, bridges, railway tracks, etc.). ) to generate the composition auxiliary line corresponding to the guide line composition method.
  • the composition methods suitable for the scene to be shot are determined to be 28/8, 3/7, 5/5, etc. , and combined with the scene to be shot to generate composition auxiliary lines corresponding to the aforementioned composition methods such as the composition of 2-8, 3-7, and 5-5.
  • the scenes to be shot corresponding to the preview images can be divided into different types, and each type of scenes to be shot can be divided into different types.
  • the composition auxiliary line corresponding to the composition mode suitable for the scene to be shot can be manually defined.
  • the matching rules described in a) to e) above are only exemplary descriptions, and the embodiments of the present application do not limit the matching rules between the shooting scene and the composition mode.
  • the cloud 220 may also include more composition methods suitable for different shooting scenarios.
  • composition guide line corresponding to the composition mode suitable for the scene to be shot generated by the cloud 220 according to the preview image can be essentially a coordinate data set composed of coordinates of multiple pixels in the preview image, and the coordinate data set The pixel points corresponding to the coordinates of the multiple pixel points in the image are connected to form the composition auxiliary line.
  • the matching rule when artificially defining matching rules, for any type of scene to be shot, if it is considered that there is only one composition method suitable for the scene to be shot, the matching rule may be: This type of scene to be photographed sets up a suitable composition method. If it is considered that the composition mode suitable for this type of scene to be photographed may include a variety of composition modes, the multiple composition modes may be set as composition modes suitable for this type of scene to be photographed in the matching rule. That is, in the above-mentioned artificially defined matching rules, each type of scene to be shot may correspond to one composition mode, or may correspond to multiple composition modes, which is not limited in this application.
  • the matching rule includes one composition method corresponding to the type of scene to be photographed
  • the cloud 220 after acquiring the preview image corresponding to the type of scene to be photographed,
  • the composition guideline corresponding to this composition method will be generated directly according to the matching rules.
  • the cloud 220 will, according to the matching rule, select images that conform to this type of image.
  • a composition mode is randomly selected among the multiple composition modes of the scene to be shot, and a composition auxiliary line corresponding to this composition mode is generated and sent to the terminal device 210 .
  • the various composition methods are set to be suitable for this type of scene to be shot.
  • the cloud 220 selects a composition method from a variety of composition methods that conform to the scene to be shot, it will select a composition method with the highest corresponding probability according to the respective probabilities of the various composition methods, and generate a composition method corresponding to this composition method.
  • the composition auxiliary line is sent to the terminal device 210 .
  • composition mode 1 accounts for The ratio is one-half, the proportion of composition mode 2 is three-eighths, and the proportion of composition mode 3 is one-eighth, then when artificially defining matching rules, you can set these three composition modes to be suitable for the different types of composition methods for the scene to be shot, but at the same time, the probability of composition method 1 is set to 1/2, the probability of composition method 2 is three-eighths, and the probability of composition method 3 is set according to the proportion of these three composition methods. The probability is one in eight.
  • the cloud selects a composition mode from composition mode 1, composition mode 2, and composition mode 3 that match the type of the scene to be shot, it will select composition mode 1 with a probability of 1/2.
  • the cloud 220 is preset with a trained artificial intelligence (AI) network.
  • the AI network has the function of generating a composition mode suitable for the scene to be shot according to the saliency results and scene information of the preview image.
  • the cloud 220 can first perform saliency detection on the preview image, and extract the saliency result of the preview image; and then input the saliency result and scene information of the preview image into the AI network.
  • the AI network can output probability distributions of multiple composition modes corresponding to the preview image (for the probability distribution, refer to the probabilities corresponding to different composition modes described in the foregoing embodiments). From the probability distribution, it can be known which composition method is more suitable for the scene to be shot corresponding to the preview image.
  • the cloud 220 can select a composition method with the highest probability from the various composition methods according to the probability distribution of the various composition methods corresponding to the preview image output by the AI network, and generate a composition auxiliary line corresponding to this composition method.
  • the probability of a composition mode that is not suitable for this type of scene to be shot is 0 or close to 0.
  • the above AI network can be obtained by collecting a large number of sample images obtained by shooting various shooting scenes, and scene information of the shooting scene corresponding to each sample image, and then using the sample images and scene information to train the neural network.
  • a large number of sample images obtained by shooting multiple shooting scenes and scene information of the shooting scene corresponding to each sample image can be obtained (for details of the scene information, please refer to the foregoing embodiment), and the saliency detection of each sample image can be performed. , to generate saliency results for each sample image.
  • the composition mode corresponding to each sample image is marked manually.
  • the saliency result of each sample image and the scene information of the corresponding shooting scene are used as input, and the composition method marked on the sample image is used as output to train the neural network to obtain an AI network.
  • the trained AI network can learn the scene information of the shooting scene corresponding to the sample image, the elements contained in the sample image, the positions and proportions of different elements, and the mapping relationship between the composition of the sample image. Therefore, the AI network has the function of generating a composition mode suitable for the scene to be shot according to the saliency result and scene information of the preview image.
  • the cloud 220 when there are multiple composition modes that conform to the scene to be shot, the cloud 220 will select a composition mode from the multiple composition modes, and generate a composition mode corresponding to this composition mode.
  • the composition auxiliary line is sent to the terminal device 210 .
  • the cloud 220 may also select at least two composition modes from the multiple composition modes conforming to the scene to be shot (for example, the selection probability is the highest. two composition modes), and generate composition auxiliary lines corresponding to the at least two composition modes described above, and send them to the terminal device 210.
  • the terminal device 210 may mainly display the composition auxiliary lines corresponding to one of the composition methods on the shooting interface, and at the same time, divide the composition auxiliary lines corresponding to other composition methods into different composition lines.
  • the screen mode, or the mode of displaying the small window, or the mode of identifying the control is displayed in the shooting interface, so that the user can actively perform the switching operation, and trigger the terminal device 210 to display the composition auxiliary line mainly displayed on the shooting interface, in the aforementioned at least two You can switch from the composition guide line corresponding to each composition mode, for example, you can switch from the composition guide line 1 to the composition guide line 2.
  • the present application does not limit the specific manner in which the terminal device 210 displays various composition auxiliary lines.
  • the terminal device 210 may correspondingly receive a composition auxiliary line once. From the second time the terminal device 210 receives the composition guideline, for each received composition guideline, the terminal device 210 displays the composition guideline received this time on the shooting interface. The last received composition guides displayed are updated.
  • the composition guide line is the cloud 220 is generated according to the preview image and scene information sent by the terminal device 210 at time (t-dt).
  • t is greater than 0
  • dt is greater than 0
  • dt represents the delay of the entire processing process from the terminal device 210 sending the preview image and scene information to the cloud 220 to the cloud 220 returning the composition auxiliary line to the terminal 210 .
  • the composition guideline displayed by the terminal device 210 may indicate to the user a composition method suitable for the scene to be photographed, and guide the user to determine the shooting position, angle, etc. of the terminal device 210 according to the composition method indicated by the composition guideline.
  • the adjustment is performed to complete the composition, so that a user without rich shooting experience can also complete a better composition when taking a photo, and the user's experience can be better.
  • the photographing method can make it easier for the user to know which composition method to use for composing a picture, and does not need to judge the composition method by himself, and does not require complicated composition operations.
  • the cloud 220 after each time the cloud 220 receives the preview image and scene information, before generating the composition guide line corresponding to the composition mode suitable for the scene to be shot according to the preview image and scene information received this time. , you can also correct the preview image first.
  • the cloud 220 can generate a composition guide line corresponding to a composition mode suitable for the scene to be shot according to the corrected preview image and scene information.
  • FIG. 2A shows a schematic flowchart of the cloud correcting the preview image in the embodiment of the present application.
  • the cloud 220 may include: a line detection network and an image correction module.
  • the correction process of the preview image by the cloud 220 may include: the cloud 220 inputs the preview image to a line detection network, and detects lines such as horizontal lines, vertical lines, and building outlines contained in the preview image through the line detection network. After the line detection network detects the lines contained in the preview image, the image correction module determines the transformation matrix required to correct the preview image according to the lines contained in the preview image, and uses the transformation matrix to correct the preview image to obtain the corrected image. Preview image.
  • the line detection network can be an end-to-end trained AI network model.
  • FIG. 2B shows a schematic structural diagram of a line detection network provided by an embodiment of the present application.
  • the line detection network may include: a backbone network, a junction proposal module, a line sampling module, and a line verification module.
  • the backbone network After inputting the preview image into the backbone network, the backbone network can extract the features in the preview image and output the shared convolution feature map corresponding to the preview image to the connection point prediction module.
  • the connection point prediction module can output the candidate connection points corresponding to the preview image according to the shared convolution feature map, and transmit it to the line segment sampling module.
  • the line segment sampling module can predict the lines (or line segments) included in the preview image according to the candidate connection points.
  • the line segment correction module can classify the predicted lines, and finally output the lines contained in the detected preview image.
  • the transformation matrix required for correcting the preview image determined by the image correction module according to the lines contained in the preview image may include, but is not limited to, a rotation matrix and a homography matrix.
  • the image correction module may determine the transformation matrix as a rotation matrix, and use the rotation matrix to transform and adjust the preview image.
  • FIG. 2C shows a schematic diagram of the effect of image correction provided by the embodiment of the present application.
  • the line detection network can detect that the horizontal line it contains (such as the boundary line between the sky and the sea) is inclined to the right, and the image correction module can determine the correctness of (a) in Figure 2C. ) to perform the required rotation matrix for correction of the preview image shown in ), and use the rotation matrix to transform and adjust the preview image.
  • the corrected preview image obtained by correcting the preview image shown in (a) in FIG. 2C may be as shown in (b) in FIG. 2C .
  • the image correction module can determine that the transformation matrix is a homography matrix, and use the homography matrix to transform and adjust the preview image.
  • FIG. 2D shows a schematic diagram of another effect of image correction provided by an embodiment of the present application.
  • the line detection network can detect that the buildings it contains have perspective problems, and the image correction module can determine the required correction for the preview image shown in (a) in Figure 2D.
  • the homography matrix of and use the homography matrix to transform and adjust the preview image.
  • the corrected preview image obtained by correcting the preview image shown in (a) in FIG. 2D may be as shown in (b) in FIG. 2D .
  • the cloud 220 corrects the preview image, so that the cloud 220 can more accurately identify the preview when generating a composition guide line corresponding to a composition method suitable for the scene to be shot according to the corrected preview image and scene information.
  • a terminal device 210 is exemplarily given in the above FIG. 2 .
  • the terminal device 210 in the terminal-cloud collaboration system may include one or more, and the multiple terminal devices 210 may be the same, different or partially the same, which are not limited herein.
  • the photographing method provided by the embodiment of the present application is a process of realizing photographing for the interaction between each terminal device 210 and the cloud 220 .
  • composition auxiliary lines described in the embodiments of the present application may also be referred to as reference lines, composition reference lines, composition lines, etc., and the names of the composition auxiliary lines are not limited herein.
  • the photographing method provided by the embodiment of the present application is exemplarily described below in combination with a scene where a user uses a mobile phone to take a photograph.
  • the embodiments of the present application are described by taking the terminal device 210 as a mobile phone as an example, it should be understood that the photographing method provided in the embodiments of the present application is also applicable to the above-mentioned other terminal devices with photographing functions.
  • the specific type of the terminal device 210 is not limited.
  • FIG. 3 shows a schematic structural diagram of a terminal device provided by an embodiment of the present application.
  • the mobile phone may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, Antenna 2, Mobile Communication Module 350, Wireless Communication Module 360, Audio Module 370, Speaker 370A, Receiver 370B, Microphone 370C, Headphone Interface 370D, Sensor Module 380, Key 390, Motor 391, Indicator 392, Camera 393, Display screen 394, and a subscriber identification module (subscriber identification module, SIM) card interface 395 and the like.
  • SIM subscriber identification module
  • the processor 310 may include one or more processing units, for example, the processor 310 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • application processor application processor, AP
  • modem processor graphics processor
  • ISP image signal processor
  • controller memory
  • video codec digital signal processor
  • DSP digital signal processor
  • NPU neural-network processing unit
  • the controller can be the nerve center and command center of the mobile phone.
  • the controller can generate the operation control signal according to the instruction operation code and the timing signal, and complete the control of reading the instruction and executing the instruction.
  • a memory may also be provided in the processor 310 for storing instructions and data.
  • the memory in processor 310 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 310 . If the processor 310 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided, and the waiting time of the processor 310 is reduced, thereby increasing the efficiency of the system.
  • processor 310 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, SIM interface, and/or USB interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transceiver
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM interface SIM interface
  • USB interface etc.
  • the external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone.
  • the external memory card communicates with the processor 310 through the external memory interface 320 to realize the data storage function. For example to save files like music, video etc in external memory card.
  • Internal memory 321 may be used to store computer executable program code, which includes instructions.
  • the processor 310 executes various functional applications and data processing of the mobile phone by executing the instructions stored in the internal memory 321 .
  • the internal memory 321 may include a storage program area and a storage data area.
  • the storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like.
  • the storage data area can store data (such as image data, phone book, etc.) created during the use of the mobile phone.
  • the internal memory 321 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like.
  • the charging management module 340 is used to receive charging input from the charger. While the charging management module 340 charges the battery 342 , it can also supply power to the mobile phone through the power management module 341 .
  • the power management module 341 is used to connect the battery 342 , the charging management module 340 , and the processor 310 .
  • the power management module 341 can also receive the input of the battery 342 to supply power to the mobile phone.
  • the wireless communication function of the mobile phone can be realized by the antenna 1, the antenna 2, the mobile communication module 350, the wireless communication module 360, the modulation and demodulation processor, the baseband processor, and the like.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in a cell phone can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
  • the antenna 1 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.
  • the mobile phone can send the obtained preview image and scene information to the cloud 220 through the wireless communication function, and receive the composition auxiliary line sent by the cloud 220 according to the preview image and the scene information.
  • the mobile phone can implement audio functions through an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, and an application processor. Such as music playback, recording, etc.
  • the sensor module 380 may include a pressure sensor 380A, a gyro sensor 380B, an air pressure sensor 380C, a magnetic sensor 380D, an acceleration sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, and an ambient light sensor 380L, bone conduction sensor 380M, etc.
  • FIG. 3 only exemplifies some of the sensors included in the sensor module 380.
  • the sensor module 380 also includes a position sensor (eg, GPS). When the mobile phone takes a picture, the sensor in the sensor module 380 can collect scene information of the scene to be photographed, such as air pressure information, temperature information, location information (such as GPS coordinates), light intensity information, and the like.
  • the camera 393 may include various types.
  • the camera 393 may include a telephoto camera with different focal lengths, a wide-angle camera or an ultra-wide-angle camera, and the like.
  • the telephoto camera has a small field of view and is suitable for shooting scenes in a small range in the distance;
  • the wide-angle camera has a larger field of view;
  • the ultra-wide-angle camera has a larger field of view than the wide-angle camera, and can be used to shoot panoramas and other large areas screen.
  • the telephoto camera with a smaller field of view can be rotated to capture scenes in different ranges.
  • the mobile phone can capture the original image (also called RAW image or digital negative) of the scene to be shot through the camera 393 .
  • the camera 393 includes at least a lens and a sensor.
  • the sensor can convert the optical signal passing through the lens into an electrical signal, and then perform analog-to-digital (A/D) conversion on the electrical signal to output a corresponding digital signal.
  • the digital signal is the RAW image.
  • the mobile phone can perform subsequent ISP processing and YUV domain processing on the RAW image through the processor (such as ISP, DSP, etc.), and convert the RAW image into an image that can be used for display, such as: JPEG image or high-efficiency image file. Format (high efficiency image file format, HEIF) image. JPEG images or HEIF images can be transmitted to the phone's display screen for display and/or to the phone's memory for storage. Thus, the mobile phone can realize the function of shooting.
  • the processor such as ISP, DSP, etc.
  • Format high efficiency image file format, HEIF
  • JPEG images or HEIF images can be transmitted to the phone's display screen for display and/or to the phone's memory for storage.
  • the mobile phone can realize the function of shooting.
  • the photosensitive element of the sensor may be a charge coupled device (CCD), and the sensor also includes an A/D converter.
  • the photosensitive element of the sensor may be a complementary metal-oxide-semiconductor (CMOS).
  • CMOS complementary metal-oxide-semiconductor
  • ISP processing may include: bad pixel correction (DPC), RAW domain noise reduction, black level correction (BLC), lens shading correction (LSC), auto white Balance (auto white balance, AWB), demosaicing (demosica) color interpolation, color correction (color correction matrix, CCM), dynamic range compression (dynamic range compression, DRC), gamma (gamma), 3D lookup table (look up) table, LUT), YUV domain noise reduction, sharpening (sharpen), detail enhancement (detail enhancement), etc.
  • DPC bad pixel correction
  • BLC black level correction
  • LSC lens shading correction
  • AWB auto white Balance
  • demosaicing demosaicing
  • CCM color correction matrix
  • DRC dynamic range compression
  • gamma gamma
  • 3D lookup table look up
  • LUT LUT
  • YUV domain noise reduction sharpening
  • sharpening sharpening
  • detail enhancement detail enhancement
  • YUV domain processing can include: multi-frame registration, fusion, noise reduction of high-dynamic range images (high-dynamic range, HDR), as well as super resolution (super resolution, SR) algorithms to improve clarity, skin beautification algorithms, distortion Correction algorithm, blur algorithm, etc.
  • high-dynamic range HDR
  • super resolution super resolution
  • Display screen 394 is used to display images, videos, and the like.
  • Display screen 394 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light).
  • LED organic light-emitting diode
  • AMOLED organic light-emitting diode
  • FLED flexible light-emitting diode
  • Miniled MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on.
  • the cell phone may include 1 or N display screens 394, where N is a positive integer greater than 1.
  • the display screen 394 may be used to display a photographing interface, a photo playing interface, and the like.
  • the shooting interface may include a preview image and a composition guideline sent by the cloud 220 to the mobile phone.
  • the mobile phone realizes the display function through the GPU, the display screen 394, and the application processor.
  • the GPU is a microprocessor for image processing, and is connected to the display screen 394 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 310 may include one or more GPUs that execute program instructions to generate or alter display information.
  • the structure shown in FIG. 3 does not constitute a specific limitation on the mobile phone.
  • the mobile phone may also include more or less components than those shown in FIG. 3, or some components may be combined, or some components may be separated, or different component arrangements, and the like.
  • some of the components shown in FIG. 3 may be implemented in hardware, software, or a combination of software and hardware.
  • terminal device 210 is other mobile terminals such as tablet computers, wearable devices, vehicle-mounted devices, AR/VR devices, notebook computers, UMPCs, netbooks, PDAs, etc., or digital cameras, SLR cameras/mirror cameras, action cameras,
  • the specific structures of these other terminal equipment can also be referred to as shown in Figure 3.
  • other terminal devices may have components added or reduced on the basis of the structure given in FIG. 3 , which will not be repeated here.
  • the terminal device 210 may run one or more photographing applications, so that the photographing function can be realized by running the photographing application.
  • the camera application may include a system level application: camera.
  • the photographing application program may also include other application programs installed in the terminal device that can be used for shooting.
  • FIG. 4 shows a schematic flowchart of a photographing method provided by an embodiment of the present application.
  • the photographing method provided by this embodiment of the present application may include: S401-S407.
  • the mobile phone After starting and running the photographing application, the mobile phone obtains a preview image corresponding to the scene to be photographed according to the first frame rate, and displays it on the photographing interface; at the same time, according to the first frequency, the sensor collects scene information of the scene to be photographed.
  • the camera application program of the mobile phone can be started first.
  • the user can click or touch the icon of the camera on the mobile phone, and the mobile phone can start the camera in response to the user's click or touch operation on the icon of the camera (or the user can also activate the camera through a voice assistant, without limitation).
  • the mobile phone can collect the RAW image corresponding to the scene to be photographed through the camera module (such as the aforementioned camera). Then, the processor of the mobile phone can perform simple ISP processing on the RAW image to obtain the YUV image corresponding to the scene to be shot.
  • the YUV image is the preview image corresponding to the scene to be shot.
  • the processor in the mobile phone may also convert the YUV image into an RGB image in RGB format.
  • the RGB image can be used as a preview image corresponding to the scene to be shot.
  • the photographing interface provided by the photographing application is displayed, and the photographing interface can be used to display the preview image of the photographing, so that the user can see the preview image of the scene to be photographed.
  • the refresh frame rate of the preview image is the first frame rate.
  • the mobile phone can collect scene information of the scene to be shot through the sensor according to the first frequency.
  • the first frequency and the specific process of collecting scene information of the scene to be shot by the sensor please refer to the description in the foregoing embodiment, and will not be repeated here.
  • the mobile phone sends the preview image and scene information to the cloud according to the second frequency.
  • the mobile phone synchronously executes S402 during the process of executing S401.
  • the cloud receives the preview image and scene information sent (or referred to as uploading) by the mobile phone.
  • the preview image sent by the mobile phone to the cloud may be the YUV image or the RGB image mentioned in the exemplary description of S401, which is not limited herein.
  • the cloud may perform the steps described in S403-S405.
  • the cloud corrects the preview image to obtain a corrected preview image.
  • S404 The cloud generates, according to the corrected preview image, a composition guide line corresponding to a composition mode suitable for the scene to be shot.
  • composition guide line corresponding to a composition mode suitable for the scene to be shot.
  • the cloud returns the composition auxiliary line to the mobile phone.
  • the mobile phone receives the composition auxiliary line returned by the cloud.
  • the composition auxiliary line returned by the cloud to the mobile phone may be a coordinate data set composed of coordinates of multiple pixels in the preview image, as described in the foregoing embodiments.
  • the cloud may also return an image containing the composition guideline to the mobile phone.
  • the pixel value of the pixel in the area where the composition guideline is located may be 0, and the pixel value of the area other than the composition guideline may be 0.
  • the pixel value of a point can be P, where P is an integer greater than 0 and less than or equal to 255.
  • the cloud can realize returning the composition auxiliary line to the mobile phone.
  • the pixel values of the pixel points of the regions other than the composition auxiliary lines may be all 255. It should be noted that this application does not limit the pixel values of the pixel points in the area where the composition auxiliary line is located and the area other than the composition auxiliary line.
  • the pixel value of the pixel point in the area where the composition auxiliary line is located can also be 0 to For other values in 255, the pixel value of the pixel point in the area where the composition auxiliary line is located may be different from the pixel value of the pixel point in the area other than the composition auxiliary line.
  • the mobile phone displays the composition auxiliary line and the preview image being displayed on the shooting interface.
  • the user Before using the mobile phone to take a photo, the user can adjust the shooting position and angle of the mobile phone by referring to the composition guideline displayed in the shooting interface, so as to complete the composition suitable for the scene to be shot.
  • FIG. 5 shows a schematic diagram of a shooting scene provided by an embodiment of the present application
  • FIG. 6 is a schematic diagram of a shooting interface provided by an embodiment of the present application.
  • the scene to be shot includes the sky and the sea, and there is a clear dividing line between the sky and the sea.
  • the mobile phone starts and runs the photographing application, it can obtain a preview image corresponding to the shooting scene shown in FIG. 5 and send it to the cloud, where the preview image can be a YUV image or an RGB image.
  • the cloud can determine, according to the preview image (corrected preview image), that the composition mode suitable for the scene to be shot shown in FIG. 5 is symmetrical
  • the composition auxiliary line corresponding to the symmetrical composition method.
  • the cloud can return the composition guideline to the mobile phone.
  • the composition guideline can be displayed on the shooting interface together with the preview image being displayed.
  • the photographing interface at least includes: a preview image presented by the preview image being displayed, and a composition auxiliary line 601 .
  • the preview image is the preview image corresponding to the scene to be shot, including the sky and the sea, and there is a clear dividing line between the sky and the sea.
  • the composition auxiliary line 601 is the composition auxiliary line corresponding to the symmetrical composition mode generated by the cloud in combination with the scene to be shot shown in FIG. 5 , and can indicate to the user that the composition mode suitable for the scene to be shot is symmetrical composition.
  • the user uses the mobile phone to take a photo, he can preview the shooting screen with reference to the preview screen displayed in the shooting interface shown in FIG. etc. to make adjustments to achieve a symmetrical composition suitable for the scene to be shot during preview.
  • FIG. 7 shows another schematic diagram of a shooting interface provided by an embodiment of the present application, and the preview screen after completing the symmetrical composition may be as shown in FIG. 7 .
  • the mobile phone may display the composition guideline together with the preview image being displayed on the shooting interface, and may also display a text prompt in the shooting interface, prompting the user to complete the composition according to the guideline of the composition guideline.
  • a text prompt may also be displayed in the shooting interface shown in FIG. 6: "Please move the boundary between the sky and the sea to coincide with the composition auxiliary line".
  • the text prompt may also be generated by the cloud when the composition auxiliary line is generated.
  • the cloud can send the composition guides and text prompts to the mobile phone together.
  • the embodiment of the present application does not limit the position of the text prompt in the shooting interface and the specific content of the text prompt. For example, when the scene to be shot is different or the composition guideline is different, the text prompts may be different.
  • the user may perform a photographing operation to trigger the mobile phone to take a photograph.
  • the photographing method further includes S407.
  • the mobile phone takes a picture in response to the user's photographing operation.
  • FIG. 8 shows a schematic diagram of a photographing operation provided by an embodiment of the present application.
  • the photographing interface provided by the mobile phone through the photographing application program may further include a photographing button 602 (not shown in the aforementioned FIGS. 6 and 7 ), and the essence of the photographing button 602 may be A function control displayed in the capture interface.
  • the user can click or touch the camera button 602, and the mobile phone can take a picture in response to the user's click or touch operation on the camera button 602, thereby obtaining a photo corresponding to the scene to be shot.
  • the function of the above-mentioned camera button 602 may also be implemented by other physical buttons on the mobile phone, which is not limited herein.
  • the photographing method provided by the embodiment of the present application can display composition guide lines in the photographing interface provided by the mobile phone in combination with the current photographing scene when the user takes a photograph with a mobile phone, so as to guide the user in composition.
  • the composition guideline can indicate the composition method suitable for the scene to be shot for the user, and guide the user to adjust the shooting position, angle, etc. of the mobile phone according to the composition method indicated by the composition guideline to complete the composition, so that users without rich shooting experience can be made. It can also complete a better composition when taking pictures, and the user experience can be better.
  • the embodiment of the present application can enable the user to more easily know which composition method to use to compose a picture according to the composition guideline recommended by the cloud when taking a photo with a mobile phone, and does not need to judge the composition method by himself, and does not require complicated composition. operate.
  • the mobile phone in S402 may not send the complete preview image to the cloud, but only send the image features of the preview image to the cloud.
  • a convolutional neural network CNN
  • the feature extraction of the preview image can be performed through the CNN to obtain the image features of the preview image.
  • the phone can then send the image features of the preview image to the cloud.
  • the cloud can identify the elements contained in the preview image according to the image features and scene information of the preview image, and record the positions and proportions of different elements in the preview image, which will not be repeated here.
  • the mobile phone Compared with the way that the mobile phone directly sends the preview image to the cloud, the mobile phone only sends the image features of the preview image to the cloud, which is used for the cloud to generate the composition guide line corresponding to the composition method suitable for the scene to be shot, which can better protect the user's privacy. Prevent user privacy leakage.
  • the composition assistant corresponding to the composition mode suitable for the scene to be shot is determined.
  • Line method when the cloud cannot determine the composition guide line corresponding to the composition method suitable for the scene to be shot according to the preset matching rules (that is, when the matching rule does not include the shooting scene corresponding to the preview image), the cloud can also Composition guides are not generated.
  • the mobile phone will not display the composition guidelines.
  • the mobile phone can display the ordinary nine-square grid auxiliary line.
  • the mobile phone also has a function of turning on and off the display of composition auxiliary lines.
  • FIG. 9 shows another schematic diagram of a photographing interface provided by an embodiment of the present application.
  • the photographing interface of the mobile phone may further include a setting button 901 , and the essence of the setting button 901 may also be a function control displayed in the photographing interface.
  • the mobile phone can switch the display interface from the photographing interface to the setting interface of the photographing application of the mobile phone in response to the user's clicking or touching operation on the setting button 901.
  • FIG. 10 shows a schematic diagram of a setting interface provided by an embodiment of the present application. As shown in FIG.
  • the setting interface at least includes: a text mark of “composition auxiliary line”, and a composition auxiliary function switch 1001 (essentially a function control) is provided on the side of the text mark.
  • the composition auxiliary function switch 1001 is composed of a sliding area and a slider (the area filled with black in the figure is the slider).
  • the mobile phone can control to turn on the display of the composition guideline in response to the user's operation of moving the slider to the right.
  • the user needs to turn off the display function of the composition guides they can click or touch the slider to move the slider to the left.
  • the mobile phone can control the display of composition guides to be turned off in response to the user's operation of moving the slider to the left.
  • the mobile phone can only send the preview image to the cloud when the display function of the composition guide line is enabled to obtain the information returned by the cloud according to the preview image.
  • Composition guideline When the display of the composition guideline is turned off, the composition guideline will not be displayed in the shooting interface every time the mobile phone takes a photo. The mobile phone will not display the composition guideline in the shooting interface until the user turns on the composition guideline display again.
  • the mobile phone may further provide the user with a function button on the shooting interface for turning off the display of the composition auxiliary line.
  • FIG. 11 shows another schematic diagram of a shooting interface provided by an embodiment of the present application.
  • the shooting interface provided by the mobile phone may also include text prompts: "Composition Guideline” 1101, and a function control after "Composition Guideline” 1101 : "X” 1102.
  • the “composition guideline” 1101 is used to prompt the user that the dotted line displayed in the shooting interface is a composition guideline.
  • the function control "X" 1102 can be used to realize the function of turning off the display of composition guide lines. For example, when the user does not like or does not need the composition guideline for composition guidance, the user can click or touch the function control "X" 1102 . The mobile phone may no longer display the composition guideline in response to the user's click or touch operation on the function control "X" 1102. For some users with rich shooting experience, they may be more looking forward to composing pictures according to their own wishes when taking pictures, and do not want to be interfered by the composition auxiliary lines. In this way, the needs of more users can be further considered, and the user's experience can be improved. experience.
  • composition guideline 1101 and the function control “X” 1102 are exemplarily displayed below the composition guideline (dotted line shown in FIG. 11 ) in the above-mentioned shooting interface shown in FIG. 11 , but The present application does not limit the display area of the “composition auxiliary line” 1101 and the function control “X” 1102 .
  • the function control "X" 1102 may also be displayed above the composition auxiliary line, in the upper right corner of the preview screen, on one side of the preview screen in the shooting interface, and so on.
  • composition auxiliary line” 1101 may not be displayed, and only the function control “X” 1102 may be displayed.
  • the function controls for realizing the function of turning off the display of the composition auxiliary lines may also be displayed in other forms of logos in the shooting interface.
  • FIG. 12 shows another schematic diagram of a shooting interface provided by an embodiment of the present application.
  • the shooting interface provided by the mobile phone may also include a function control: “composition assist” 1201 .
  • composition assist When the user needs to turn off the display function of the composition auxiliary line, the user can click or touch the function control "composition auxiliary" 1201 .
  • the mobile phone can control to close the display of the composition auxiliary line in response to the user's click or touch operation on the function control “composition auxiliary” 1201 .
  • the mobile phone can also control to enable the display of composition guide lines in response to the user's click or touch operation on the function control "Composition Assistant” 1201 .
  • the function controls for enabling and disabling the display of composition auxiliary lines may also be implemented in the form of other physical buttons (different from virtual buttons in the shooting interface).
  • the camera application of the mobile phone can be configured so that a certain physical button (eg "volume +" or "volume -") on the mobile phone has the function of enabling and disabling the display of the composition auxiliary line.
  • composition auxiliary lines are all exemplary descriptions, which are not limited in this application.
  • the cloud when the cloud generates a composition guide line corresponding to a composition mode suitable for the scene to be shot according to the preview image and scene information, the cloud can also generate a shooting contour line suitable for the scene to be shot according to the preview image.
  • the cloud can further outline the elements in the preview image after generating the composition guide line corresponding to the composition method suitable for the scene to be shot (such as: building , mountain, road, etc.), and according to the composition guide line corresponding to the composition method suitable for the scene to be shot, move the contour line of the element to a position that can satisfy the composition method corresponding to the composition guide line to obtain the shooting contour line.
  • the shooting outline is generated from the composition guideline and the outline of the element in the preview image.
  • the cloud can send the composition guideline and the shooting contour line to the mobile phone at the same time, and the mobile phone can display the preview image, the composition guideline, and the shooting contour line on the shooting interface.
  • the user can refer to the composition guide displayed on the shooting interface, and when adjusting the shooting position and angle of the mobile phone, he can also refer to the shooting outline displayed on the shooting interface, and move the elements in the preview screen to the shooting position. the position indicated by the outline.
  • FIG. 13 shows another schematic diagram of a photographing interface provided by an embodiment of the present application.
  • the composition guideline displayed on the shooting interface of the mobile phone is the composition guideline 1301 corresponding to the three-point composition shown in FIG. 13 (FIG. 13 ).
  • the two vertical lines shown in ), the shooting contour line may be the contour line 1302 generated according to the contour line of the iron tower.
  • the user previews the shooting screen before taking a photo with a mobile phone he can refer to the composition auxiliary line 1301 displayed in the shooting interface shown in FIG. ), and at the same time, referring to the shooting contour line 1302, move the iron tower to the area where the shooting contour line 1302 is located. In this way, the user's operation of composing pictures can be further simplified, and the user experience can be improved.
  • the shooting contour line returned from the cloud to the mobile phone may also be a coordinate data set composed of multiple pixel coordinates in the preview image (different from the coordinate data set of the composition auxiliary line) .
  • the cloud can also return to the mobile phone an image containing the composition guideline and the shooting contour line.
  • the pixel value of the pixel in the area where the composition guideline is located can be 0; the pixel value of the pixel in the area where the shooting contour line is located The value can also be 0.
  • the pixel values of the pixel points of the regions other than the composition auxiliary line and the shooting contour line may all be P, and P is an integer greater than 0 and less than or equal to 255.
  • the pixel values of the pixel points of the regions other than the composition auxiliary lines may be all 255.
  • this application does not limit the pixel value of the pixel in the area where the contour line is shot, and the pixel value of the pixel in the area where the contour line is shot is the same as the pixel value in the area other than the composition auxiliary line and the contour line. It is sufficient that the pixel values are different, and the pixel values of the pixel points in the area where the contour line is located and the pixel values of the pixel points in the area where the composition auxiliary line is located may be the same or different.
  • the mobile phone takes a photograph in response to a user operation, and after obtaining the photograph corresponding to the scene to be photographed, the photograph can also be uploaded to the cloud.
  • the cloud can optimize the composition of the photo, and return the photo with the optimized composition to the mobile phone for display, so that the user can choose whether to save it.
  • FIG. 14 shows another schematic flowchart of the photographing method provided by the embodiment of the present application.
  • the photographing method provided by this embodiment of the present application may further include S1401 to S1404 on the basis of the foregoing shown in FIG. 4 .
  • the mobile phone sends an initial photo obtained by taking a picture of the scene to be shot to the cloud.
  • the mobile phone can obtain a photo corresponding to the scene to be shot, and the photo can be referred to as an initial photo.
  • the cloud receives the initial photo uploaded from the phone.
  • the cloud optimizes the composition of the initial photo to obtain at least one photo with the optimized composition.
  • the cloud returns at least one photo with optimized composition to the mobile phone.
  • the mobile phone receives the composition-optimized photo from the cloud.
  • the mobile phone displays an initial photo and at least one photo whose composition is optimized.
  • the mobile phone may display the photo with optimized composition and the initial photo taken by the user, for the user to select one or more of them to save.
  • composition optimization process shown in FIG. 14 is exemplified in more detail below.
  • FIG. 15 shows a schematic flowchart of the composition optimization of the initial photo by the cloud in the embodiment of the present application.
  • the cloud may at least include: a line detection network, an image correction module, a saliency detection module, and an aesthetic scoring network.
  • the cloud can detect the horizontal lines, vertical lines, building outlines and other lines contained in the initial photo through the line detection network.
  • the image correction module can determine the transformation matrix required to correct the initial photo according to the lines contained in the initial photo, and use the transformation matrix to correct the initial photo.
  • the saliency detection module can perform saliency detection on the corrected photo to obtain the saliency result in the photo.
  • the cloud can cut the corrected photos according to the aesthetic scoring network and the aforementioned saliency results of the corrected photos to obtain multiple candidate cropping images. And score multiple candidate crop images. Then, the cloud can send at least one candidate cropping image with a high score in the multiple candidate cropping images (for example: one or two with the highest ranking) as a composition-optimized photo to the mobile phone.
  • the line detection network and the image correction module described herein may be the same as the line detection network and the image correction module shown in the aforementioned FIG. 2A .
  • the specific composition of the line detection network can be referred to as shown in the aforementioned FIG. 2B .
  • the process that the cloud detects horizontal lines, vertical lines, building outlines and other lines contained in the initial photo through the line detection network may include: after the initial photo is input into the backbone network, the backbone network extracts the features in the initial photo, and outputs the initial
  • the shared convolutional feature map corresponding to the photo is connected to the point prediction module.
  • the connection point prediction module outputs the candidate connection points corresponding to the initial photo according to the shared convolution feature map, and transmits it to the line segment sampling module.
  • the line segment sampling module predicts the lines (or line segments) contained in the initial photo according to the candidate connection points.
  • the line segment correction module classifies the predicted lines and finally outputs the detected lines contained in the initial photo.
  • the image correction module determines the effect of correcting the initial photo according to the lines contained in the initial photo, and can refer to the effect of correcting the preview image shown in the aforementioned FIG. 2C and FIG. 2D .
  • the correction principle is the same and will not be repeated.
  • the saliency detection module can use the Mask RCNN segmentation method to segment the human body, animals, buildings and other points of interest in the corrected photo as saliency results.
  • the saliency detection module can segment the corrected photos that are related to aesthetic composition.
  • elements related to aesthetic composition can be dividing lines, roads, bridges, etc.
  • the saliency detection module can segment the saliency results in the photo according to the saliency elements defined by humans, which is not limited herein.
  • the aesthetic scoring network may at least include: a trained object detection (single shot multiBox detector) SSD network and a regression model.
  • the cloud can input the corrected photos and the saliency results of the corrected photos into the SSD network, and calculate the corrected photos according to the preset multiple cropping frames (which can be called the first cropping frame) through the SSD network.
  • the number of preset cropping boxes is not limited, and can be defined or configured manually.
  • the cloud can sort the preset multiple trimming boxes according to the scores corresponding to the multiple trimming boxes output by the SSD network, and determine the trimming box with the highest score.
  • the cloud can input the corrected photo, the saliency result of the corrected photo, and the cropping box with the highest score obtained through the SSD network into the regression model for regression, and obtain multiple final cropping boxes (which can be called the first cropping box). two crop boxes) and the corresponding scores.
  • the cloud can crop the corrected photo according to multiple final cropping frames to obtain multiple candidate cropping images, and the score corresponding to each candidate cropping image is the score corresponding to the final cropping frame.
  • FIG. 16 shows a schematic diagram of the regression principle provided by the embodiment of the present application.
  • the size of the corrected photo is m1*n1 (that is, it contains m1*n1 pixels, and each small square in FIG. 16 can be regarded as a pixel).
  • the cropped box with the highest score obtained by the SSD network is the rectangular box shown by the thick solid line in Figure 16.
  • the coordinates of the upper left corner pixel point a of the rectangular frame are (x1, y1), and the coordinates of the lower right corner pixel point b are (x2, y2).
  • the connecting line can be used as a diagonal line to divide a rectangular area (hereinafter referred to as the first area) in the corrected photo, that is, The upper left corner of the first area is the upper left corner of the corrected photo, and the lower right corner is the pixel point a.
  • the connecting line can also be used as a diagonal line to divide a rectangular area (hereinafter referred to as the second area) in the corrected photo, that is, , the upper left corner of the second area is the pixel point b, and the lower right corner is the lower right corner of the corrected photo.
  • the upper left corner of the cropping frame can be limited to the aforementioned first area, and the lower right corner can be restricted to the aforementioned second area for regression.
  • the crop box may be shown as a number of dashed boxes in FIG. 16 .
  • the first area and the second area have the same size, which is m2*n2. It can be understood that m2 is smaller than m1, n2 is smaller than n1, and m1, m2, n1, and n2 are all positive integers.
  • the above-mentioned SSD network can be obtained by training with some public datasets, such as: a comparative photo composition dataset (CPC).
  • CPC comparative photo composition dataset
  • the CPC dataset includes multiple photos (or pictures), and each photo is marked with corresponding aesthetic scores when different cropping methods are used.
  • Mask RCNN can be used to segment the photos in the CPC dataset first, and the saliency results corresponding to each photo can be obtained. Then, each photo and the corresponding saliency results are input into the SSD network for training, so that the trained SSD network can score crop boxes of different scales.
  • the cloud can use at least one candidate cropping image with a high score in the multiple candidate cropping images (such as the one or two with the highest ranking) as the composition.
  • the optimized photo is sent to the phone.
  • the mobile phone After receiving the composition-optimized photo from the cloud, the mobile phone can display the initial photo and the composition-optimized photo for the user to choose whether to save or not. For example, the user may select the first photo to save, and the first photo may be one or more of the initial photo and the photo after composition optimization.
  • FIG. 17 shows a schematic diagram of composition optimization provided by an embodiment of the present application.
  • the initial photo corresponding to the scene to be shot obtained by taking a picture of the mobile phone is shown in (a) of FIG. 17
  • the photo after image correction of the initial photo by the cloud may be shown in (b) of FIG. 17 .
  • the saliency result obtained by performing saliency detection on the photo shown in (b) in Figure 17 in the cloud can be shown in (c) in Figure 17, including a horseman and a lighthouse.
  • the cloud can input the photos shown in (b) in Figure 17 and the saliency results shown in (c) in Figure 17 into the aesthetic scoring network to obtain multiple candidate cropping maps output by the aesthetic scoring network , and select the two highest rated photos from the multiple candidate crop images as the composition-optimized photos.
  • the composition-optimized photo may be shown in (d) and (e) of FIG. 17 .
  • FIG. 18 shows a schematic diagram of a photo display interface provided by an embodiment of the present application.
  • the mobile phone may display the initial photo shown in (a) in FIG. 17 through the photo display interface shown in FIG. 18 for the user to check the photographing effect.
  • the photo display interface may be a playback interface of the gallery of the mobile phone.
  • the mobile phone can also display the received two photos with optimized composition as shown in (d) and (e) of Fig. 17 below the original photo.
  • the top of the initial photo also includes a selection function control "A" 1801, and the upper parts of the two composition-optimized photos respectively include a selection function control "B1" 1802 and a selection function control "B2" 1803.
  • the photo display interface also includes a confirm button 1804 (check mark shown in FIG. 18 ) and a cancel button 1805 (cross mark shown in FIG. 18 ). Both confirm button 1804 and cancel button 1805 are functional controls with different functions.
  • the mobile phone can respond to the user's click on the corresponding function control. Click or touch to select the corresponding photo. Then, when the user clicks or touches the confirmation button 1804 below, the mobile phone can save the aforementioned selected photo in response to the user's click or touch operation on the confirmation button 1804 . For example, when the user wants to save the photo corresponding to the function control "B2" 1803, he can click to select the function control "B2" 1803, select the photo, and then click the OK button 1804 below to save the photo.
  • the mobile phone can switch to the shooting interface again for the user to take a photo.
  • the mobile phone may only save the initial photos.
  • the cloud performs further composition optimization on the initial photo taken by the mobile phone, and returns the optimized photo to the mobile phone, which can improve the success rate of composition, and the photo after composition optimization will not have distortions such as rotation and perspective. Able to provide users with better photo selection.
  • the mobile phone simultaneously displays the initial photo taken by the user and the photo with optimized composition, which can take into account the needs of the user, so that the user can choose either the photo taken by himself or the photo with optimized composition recommended by the cloud.
  • the mobile phone may first detect whether the initial photo is a portrait photo, and if the initial photo is a portrait photo, the mobile phone may directly save the portrait photo without executing the step of S1401. That is, the mobile phone does not send the initial photo to the cloud, and the cloud does not perform subsequent composition optimization on the initial photo. If the initial photo is not a portrait photo, the mobile phone can continue to perform the steps of S1401 according to the process shown in FIG. 14 , and send the initial photo to the cloud, and the cloud performs subsequent composition optimization on the initial photo.
  • the mobile phone When the mobile phone detects that the initial photo is a portrait photo, it does not send the initial photo to the cloud for composition optimization, which can better protect user privacy and prevent user privacy from being leaked.
  • FIG. 22 shows another schematic flowchart of the photographing method provided by the embodiment of the present application.
  • the photographing method provided by this embodiment of the present application may further include steps S2201 to S2203 on the basis of the aforementioned FIG. 14 .
  • the mobile phone selects the first photo in response to the user's selection operation.
  • the first photo includes one or more of an initial photo and at least one composition-optimized photo.
  • the mobile phone simultaneously displays the initial photo taken by the user and at least one photo whose composition has been optimized, and the user selects a photo that is satisfied with the composition to be saved. It is understandable that the user can choose either a photo taken by himself or a photo with an optimized composition recommended by the cloud. Optionally, the user may select one photo from the initial photo and the at least one composition-optimized photo for saving, or may select multiple photos for saving. The one or more photos saved above are the first photos.
  • the mobile phone sends the first photo to the cloud.
  • the cloud receives the first photo sent by the mobile phone.
  • the cloud may acquire the user's aesthetic preference according to the first photo.
  • the cloud can train the aesthetic scoring network in the cloud according to the first photo.
  • the aesthetic scoring network can at least include a target detection (single shot multiBox detector, SSD) network and a regression model.
  • the cloud detects the saliency of the first photo through a saliency detection module, and generates a saliency result of each first photo.
  • the cloud can then take the saliency results of each first photo as input, and the composition of the first photo as output to train the aesthetic scoring network.
  • the trained aesthetic scoring network can learn the mapping relationship between the elements contained in the first photo, the positions and proportions of different elements, and the composition of the first photo. In this way, the cloud can obtain the user's aesthetic preference and generate a composition method suitable for the scene to be shot.
  • the cloud to train the aesthetic scoring network may be the cloud to train the SSD network, that is, the cloud inputs the saliency results corresponding to each first photo into the SSD network for training, so that the trained SSD network can give different results.
  • the scale of the crop box is scored, and the ranking can be given according to the composition of the scene to be shot.
  • the cloud detects the saliency of the first photo through a saliency detection module, and generates a saliency result of each first photo. Then, the cloud can use the saliency result of each first photo and the scene information of the corresponding shooting scene as input, and the composition method of the first photo as output to train the aesthetic scoring network.
  • the trained aesthetic scoring network can learn the scene information of the shooting scene corresponding to the first photo, the elements contained in the first photo, and the mapping relationship between the positions and proportions of different elements and the composition of the first photo. In this way, the cloud can obtain the user's aesthetic preference and generate a composition method suitable for the scene to be shot.
  • the cloud to train the aesthetic scoring network may be the cloud to train the SSD network, that is, the cloud inputs the saliency results corresponding to each first photo into the SSD network for training, so that the trained SSD network can give different results.
  • the scale of the crop box is scored, and the ranking can be given according to the composition of the scene to be shot.
  • the scene information of the shooting scene corresponding to the first photo may be the scene information that has been uploaded to the cloud by the mobile phone in step S401. Describe the scene information corresponding to the first photo.
  • the scene information corresponding to the shooting scene may be sent to the cloud together.
  • the trained aesthetic scoring network can be migrated to the cloud storage space corresponding to the user, so as to obtain the user's aesthetic preference.
  • the cloud may replace the existing user-customized aesthetics scoring network with the trained aesthetics scoring network.
  • the terminal device uploads the user's selection result of photographing composition to the cloud, and the cloud trains the aesthetic scoring network to obtain the user's aesthetic preference.
  • the cloud trains the aesthetic scoring network to obtain the user's aesthetic preference.
  • the mobile phone may send the image features of the first photo to the cloud.
  • a convolutional neural network CNN may be preset in the mobile phone.
  • the mobile phone may perform feature extraction on the first photo through CNN to obtain the image features of the first photo.
  • the mobile phone can send the image features of the first photo to the cloud.
  • the cloud can identify the elements contained in the first photo according to the image features and scene information of the first photo, and record the positions and proportions of different elements in the first photo, which will not be repeated.
  • the mobile phone Compared with the way that the mobile phone directly sends the first photo to the cloud, the mobile phone only sends the image features of the first photo to the cloud, which is used for the cloud to obtain the user's aesthetic preference, which can better protect the user's privacy and prevent the leakage of user privacy.
  • the mobile phone may send the identification information of the first photo to the cloud.
  • the identification information of the first photo may be the file name of the first photo, or may be a unique identifier assigned to the first photo by the system.
  • the identification information may also be a photo hash value calculated based on the first photo.
  • the hash value is a hash value calculated by the cloud for the photo corresponding to the aforementioned scene to be shot and at least one photo whose composition is optimized, and can be used to uniquely mark the photo. It can be understood that an existing hash value algorithm in the art may be used to calculate the hash value, and details are not described herein again.
  • the cloud After receiving the identification information of the first photo sent by the mobile phone, the cloud can query the corresponding first photo stored in the cloud according to the identification information of the first photo, so that the cloud can obtain the user's aesthetic preference according to the corresponding first photo.
  • the mobile phone Compared with the way that the mobile phone directly sends the first photo to the cloud, the mobile phone sends the identification information of the first photo to the cloud, which is used for the cloud to obtain the user's aesthetic preference, which can better protect the user's privacy, prevent the leakage of user privacy, and reduce the number of mobile phones.
  • the amount of communication data with the cloud improves the efficiency of mobile phone data transmission.
  • the mobile phone sends the acquired preview image and scene information to the cloud according to the second frequency as an example for description.
  • the frequency of sending the scene information to the cloud by the mobile phone may also be different from the frequency of sending the preview image to the cloud.
  • the mobile phone can send the obtained preview image to the cloud according to the third frequency, and send the obtained scene information to the cloud according to the fourth frequency.
  • the third frequency can be smaller or larger than the fourth frequency, which is not limited here.
  • FIG. 23 shows another schematic flowchart of the photographing method provided by the embodiment of the present application. As shown in FIG. 23 , the photographing method provided by this embodiment of the present application includes steps S2301-S2308.
  • the mobile phone obtains the photo to be edited and scene information corresponding to the photo to be edited.
  • the photo to be edited may be a photo obtained by taking a photo with a mobile phone
  • the scene information corresponding to the photo to be edited may be scene information collected by the mobile phone when taking a photo.
  • the mobile phone can display the thumbnail of the photographed photograph on the editing interface, so that the user can browse and/or edit the photographed photograph.
  • the photo to be edited may be a user browsing and selecting a photo of interest in the photo album APP of the mobile phone
  • the scene information corresponding to the photo to be edited may be the scene corresponding to the shooting scene collected by the mobile phone when the photo to be edited is taken. information.
  • the mobile phone can display the thumbnail of the photo on the editing interface.
  • the above to-be-edited photos and scene information are stored in the mobile phone.
  • the scene information can be saved together with the photo to be edited, or can be saved separately from the photo to be edited.
  • the scene information may be stored in an electronic file of the photo to be edited, such as an image file in a format such as RAW, JPG, or IMG, which is not limited in this application.
  • the mobile phone After selecting the photo to be edited, the mobile phone extracts scene information from the photo to be edited. It can be understood that the mobile phone can also send the photo to be edited including the scene information to the cloud, and the cloud extracts the scene information from the photo to be edited.
  • the scene information may be stored in a photo attribute file corresponding to the photo to be edited, such as an Extensible Markup Language (Extensible Markup Language, XML) format file.
  • XML Extensible Markup Language
  • the photo attribute file can be sent to the cloud. It can be understood that the mobile phone can also store scene information in other existing ways, which is not limited in this application.
  • the mobile phone sends the photo to be edited and scene information corresponding to the photo to be edited to the cloud.
  • the mobile phone can obtain the photo to be edited and the scene information corresponding to the photo to be edited, and send the photo to be edited and the scene information corresponding to the photo to be edited to the cloud.
  • the cloud receives the photos to be edited and the corresponding scene information uploaded from the mobile phone.
  • the mobile phone may send the photo to be edited and scene information corresponding to the photo to be edited to the cloud in response to the user's editing operation.
  • the user may edit the photo to be edited in the photo editing interface of the mobile phone.
  • FIG. 24 shows a schematic diagram of a photo editing interface provided by an embodiment of the present application.
  • the photo editing interface displays a preview of the photo to be edited, and a text prompt 2401 and composition optimization function controls 2402 and 2403 may be displayed below the preview of the photo to be edited.
  • the text prompt 2401 can display the words composition optimization, for example, and the composition optimization function controls 2402 and 2403 are respectively used to cancel the operation and confirm the operation.
  • the mobile phone can activate the composition optimization function in response to the user's click or touch operation, and send the photo to be edited and the scene information corresponding to the photo to be edited to the cloud.
  • the mobile phone may not enable the composition optimization function.
  • the cloud performs composition optimization on the photo to be edited, and obtains at least one photo whose composition is optimized.
  • the cloud may at least include: a line detection network, an image correction module, a saliency detection module, and an aesthetic scoring network.
  • the cloud adopts the line detection network, the image correction module and the saliency detection module to process the photos to be edited, which can be performed with reference to the content described above, and will not be repeated here.
  • the cloud can input the corrected photo to be edited, the saliency result of the corrected photo, and the scene information into the aesthetic scoring network to obtain one or more candidate crop images as the composition-optimized photo.
  • the aesthetic scoring network may at least include: a trained SSD network and a regression model.
  • the cloud can input the corrected photo, the saliency result of the corrected photo, and the scene information into the SSD network, and calculate the corrected photo according to the preset multiple cropping frames (which can be called the first cropping frame) through the SSD network. ) corresponding to each cropping frame when cropping.
  • the number of preset cropping boxes is not limited, and can be defined or configured manually.
  • the cloud can sort the preset multiple trimming boxes according to the scores corresponding to the multiple trimming boxes output by the SSD network, and determine the trimming box with the highest score.
  • the cloud can input the corrected photo, the saliency result of the corrected photo, and the cropping box with the highest score obtained through the SSD network into the regression model for regression, and obtain multiple final cropping boxes (which can be called the first cropping box).
  • the cloud can crop the corrected photo according to multiple final cropping frames to obtain multiple candidate cropping images, and the score corresponding to each candidate cropping image is the score corresponding to the final cropping frame.
  • the regression model may adopt the regression model described above, which will not be repeated here.
  • the above-mentioned SSD network can be obtained by training with some public datasets, such as: a comparative photo composition dataset (CPC).
  • CPC comparative photo composition dataset
  • the CPC dataset includes multiple photos (or pictures), and each photo is marked with corresponding aesthetic scores when different cropping methods are used.
  • Mask RCNN can be used to segment the photos in the CPC dataset first, and the saliency results corresponding to each photo can be obtained. Then, each photo, the saliency result corresponding to the photo, and the scene information are input into the SSD network for training, so that the trained SSD network can score crop boxes of different scales.
  • the aesthetics scoring network in the cloud can be a general aesthetics scoring network, that is, the aesthetics scoring network can be used by any user.
  • the aesthetics scoring network in the cloud may also be a user-customized aesthetics scoring network corresponding to the user who uploads the photo to be edited, and the user-customized aesthetics scoring network may be stored in a cloud storage space corresponding to the user.
  • the cloud when the cloud only stores the general aesthetic scoring network, for example, the user uses the composition optimization function for the first time, the cloud uses the general aesthetic scoring network to optimize the composition of the photo to be edited to obtain at least one composition-optimized photo.
  • the cloud uses the user-customized aesthetic scoring network to optimize the composition of the photo to be edited to obtain at least one composition-optimized photo.
  • the cloud returns at least one photo with optimized composition to the mobile phone.
  • the mobile phone receives at least one composition-optimized photo from the cloud.
  • the mobile phone displays an initial photo and at least one photo whose composition is optimized.
  • the mobile phone may display the photo with optimized composition and the initial photo taken by the user, for the user to select one or more of them to save.
  • the mobile phone saves the first photo in response to the user's save operation.
  • the first photo includes one or more of an initial photo and at least one composition-optimized photo.
  • the mobile phone sends the saved first photo to the cloud.
  • the cloud receives the first photo uploaded by the mobile phone.
  • the cloud may acquire the user's aesthetic preference according to the saved first photo.
  • step S2308 in the case that the user's aesthetic preference has been stored in the cloud, after the cloud obtains the user's aesthetic preference, the existing user's aesthetic preference may be updated.
  • the terminal device uploads the photo or the existing photo taken by the user to the cloud, the cloud returns the composition-optimized photo to the terminal device based on the aesthetic scoring network, and the terminal device uploads the user's selection result of the photo composition to the cloud, and the cloud Train the aesthetic scoring network to obtain the user's aesthetic preference.
  • This device-cloud collaborative method of private customization of photo composition can rely on the strong computing power of the cloud to effectively reduce the hardware requirements of terminal equipment, continue to recommend photo composition that is more in line with user habits, reduce user learning costs, and improve user photo composition experience.
  • the terminal device may first detect whether the first photo is a portrait photo. If the first photo is a portrait photo, the terminal device may directly save the portrait photo without sending the first photo to the cloud. A photo to better protect user privacy and prevent user privacy leakage.
  • the mobile phone may not send the complete first photo to the cloud, but only send the image features of the first photo to the cloud.
  • the mobile phone may not send the complete first photo to the cloud, but only send the image features of the first photo to the cloud.
  • the mobile phone only sends the image features of the first photo to the cloud, which is used for the cloud to obtain the user's aesthetic preference, which can better protect the user's privacy and prevent the leakage of user privacy.
  • the mobile phone may send the identification information of the first photo to the cloud.
  • the mobile phone may send the identification information of the first photo to the cloud.
  • the mobile phone sends the identification information of the first photo to the cloud, which is used for the cloud to obtain the user's aesthetic preference, which can better protect the user's privacy, prevent the leakage of user privacy, and reduce the number of mobile phones.
  • the amount of communication data with the cloud improves the efficiency of mobile phone data transmission.
  • FIG. 25 shows another schematic flowchart of the photographing method provided by the embodiment of the present application. As shown in FIG. 25 , the photographing method provided by this embodiment of the present application includes steps S2501-S2507.
  • the mobile phone obtains the photo to be edited.
  • the photo to be edited may be a photo obtained by taking a photo with a mobile phone. After the user uses the mobile phone to take a picture, the mobile phone can display the thumbnail of the photographed photograph on the editing interface, so that the user can browse and/or edit the photographed photograph.
  • the photo to be edited by taking the photo above reference may be made to the photographing process described above, which will not be repeated here.
  • the photo to be edited may be the thumbnail of the photo that the user browses in the photo album APP of the mobile phone, and then a photo of interest is taken as the photo to be edited.
  • Photos to be edited can be stored on your phone, or in the cloud or on a server. After the user selects the above photo, the mobile phone can display the thumbnail of the photo on the editing interface.
  • the mobile phone acquires the edited photo in response to the user's editing operation of the photo to be edited.
  • the user's editing operation can be performed in the photo editing interface of the mobile phone.
  • the mobile phone in response to an editing operation of the photo to be edited by the user, the mobile phone may obtain the intermediate photo according to a locally saved basic image quality algorithm.
  • the user can further manually edit the intermediate photo, adjust or set photo parameters, so as to obtain an edited photo that conforms to the user's aesthetic preference.
  • Basic image quality algorithms may include beauty algorithms, photo style algorithms, filter algorithms, exposure algorithms, noise reduction algorithms, and so on.
  • the mobile phone can provide a variety of functions, such as skin resurfacing, face reduction, big eyes, thin nose, long nose, chin, forehead, as well as the functions of opening the corners of the eyes, mouth shape, and smiling mouth corners.
  • the portrait in the photo may be automatically beautified, such as whitening the skin of the portrait, resurfacing the skin, and the like.
  • the photo when an automatic exposure algorithm is applied to a photo to be edited, the photo may be automatically exposed, such as adjusting parameters such as brightness and contrast of the photo, so as to better and clearly present the content of the photo.
  • filter effects when applying a filter algorithm to a photo to be edited, filter effects such as old photo, movie style, black and white, etc. may be applied to the photo.
  • Photo parameters may include parameters such as brightness, contrast, saturation, sharpness, sharpness, noise, highlights (or “brights” or “highlights”), shadows (or “darks”), or hue.
  • the photo parameters can be stored in the photo file or in the photo attribute file corresponding to the photo file.
  • For the photo attribute file reference may be made to the foregoing description, which will not be repeated here.
  • Each basic image quality algorithm includes a corresponding basic image quality algorithm parameter, and the basic image quality algorithm parameter is used to determine the degree or intensity of applying the above-mentioned photo parameters to the photo. Based on the corresponding basic image quality algorithm parameters, photo editing operations such as beauty, photo style, filter, exposure, noise reduction, etc. can be implemented.
  • 26A to 26C show another schematic diagram of a photo editing interface provided by an embodiment of the present application.
  • the photo editing interface displays a preview of the photo to be edited, and functional controls can be displayed below the preview, such as beauty control 2601, filter control 2602, one-key optimization control 2603, and more controls 2604.
  • the photo to be edited is, for example, a portrait with some wrinkles and spots on the face.
  • the user can click the beauty control 2601 to beautify the photo to be edited.
  • FIG. 26A shows a portrait with some wrinkles and spots on the face.
  • the mobile phone beautifies the photo to be edited, and displays a preview image of the beautified intermediate photo on the photo editing interface.
  • the beauty algorithm applies a set of photo parameters to the photo to be edited, such as brightness, contrast, hue, noise removal, and speckle removal.
  • the beautifying intensity control button 2605 is displayed at the bottom of the preview image.
  • the control button 2605 is in the middle position with the parameter value of 5. As mentioned earlier, this process reflects the effect of the basic image quality algorithm parameters on the edited photo. Due to different aesthetic preferences of users, the user may think that the beauty level is too weak or too strong, and the user can further adjust the control button 2605 to weaken or enhance the beauty level.
  • the middle photo still has wrinkles (spots have been removed by beauty).
  • the user can drag the control button 2605 to the right, for example, to adjust the parameter value 7 as shown in FIG. 26C, thereby eliminating wrinkles and spots in the portrait photo.
  • a preview of the edited photo is displayed in the editing interface.
  • FIGS. 27A to 27C show another schematic diagram of a photo editing interface provided by an embodiment of the present application.
  • the photo editing interface displays a preview of the photo to be edited, and functional controls such as beauty control 2701, filter control 2702, one-key optimization control 2703, and more controls 2704 can be displayed below the preview image.
  • the photo to be edited is, for example, a landscape, the brightness of the photo is dark, and the contrast is also weak.
  • the user can click the one-click optimization control 2703 to optimize the photo to be edited.
  • the mobile phone optimizes the photo to be edited, and displays the optimized intermediate photo preview on the photo editing interface.
  • the optimization process applies a set of photo parameters such as brightness, contrast, saturation, sharpness, etc.
  • a brightness control 2705, a contrast control 2706, a saturation control 2707, and a sharpness control 2708 are displayed, respectively. Users can adjust or set the corresponding brightness, contrast, saturation and sharpness values through the above-mentioned controls.
  • the brightness control 2601 as an example, when the user drags the control to the left, the brightness value decreases and the brightness of the photo decreases; when the user drags the control to the right, the brightness value increases and the brightness of the photo increases.
  • the above controls are all in the middle of the parameter value of 0. The user makes adjustments to the intermediate photo based on aesthetic preferences.
  • the middle photo is still dark and the contrast is weak. Users can drag the above controls to adjust or set corresponding parameters to edit photos. As shown in FIG. 27C , the user adjusts the brightness value to 1, the contrast value to 2, the saturation value to remain unchanged, and the sharpness value to 1 to obtain an edited photo. A preview of the edited photo is displayed in the photo editing interface.
  • the mobile phone after the mobile phone obtains the intermediate photo according to the locally saved basic image quality algorithm, if the user is satisfied with the intermediate photo, the intermediate photo may not be edited, and the mobile phone regards the intermediate photo as an edited photo. In other words, if the intermediate photo conforms to the user's aesthetic preference, the mobile phone can use the intermediate photo as the input data for training the cloud-based photographic style optimization network in order to obtain the user's aesthetic preference.
  • the editing interface and function controls may also adopt other existing display styles or arrangement manners, so as to realize the corresponding adjustment of the photos, which is not limited in this application.
  • the mobile phone sends the photo to be edited and the edited photo to the cloud.
  • the cloud receives the photos to be edited and the edited photos uploaded from the mobile phone.
  • the cloud obtains the user's aesthetic preference according to the photo to be edited and the edited photo.
  • the cloud obtains basic image quality optimization parameters according to the user's aesthetic preference.
  • the cloud returns basic image quality optimization parameters to the mobile phone.
  • the mobile phone receives the basic image quality optimization parameters from the cloud.
  • the mobile phone updates the local basic image quality algorithm according to the basic image quality optimization parameters.
  • the mobile phone can replace the basic image quality parameters stored locally on the mobile phone, that is, the mobile phone updates the local basic image quality algorithm through the basic image quality optimization parameters.
  • the mobile phone processes photos through the basic image quality algorithm, it can obtain photos with a photo-taking style that is more in line with the user's habits, improving the user's photo-taking experience.
  • the cloud may model the basic image quality algorithm to obtain a relationship function (model) between the parameters of the basic image quality algorithm and the imaging effect.
  • the parameters of the basic image quality algorithm may be preliminarily determined based on the existing basic image quality algorithm.
  • the parameters of the basic image quality algorithm can be preliminarily determined based on the existing average brightness method, and the cloud can establish a relationship function between the parameters of the automatic exposure algorithm and the brightness imaging effect.
  • the basic image quality algorithm parameters can be preliminarily determined based on the existing Gaussian denoising method, and the cloud can establish a relationship function between the noise reduction algorithm parameters and the noise reduction imaging effect.
  • the skin color region is smoothed based on the bilateral filtering method, and then the color tone is adjusted according to the skin color, and the cloud can establish a relationship function between the parameters of the beauty algorithm and the beauty imaging effect.
  • the basic image quality algorithm also includes a saturation algorithm, a hue adjustment algorithm, and the like, which are not limited in this application.
  • the cloud can train the aesthetic scoring network based on the photos to be edited and the edited photos. Specifically, the cloud can train the aforementioned relationship function between the parameters of the basic image quality algorithm and the imaging effect, and migrate the trained relationship function to the cloud storage space corresponding to the user to obtain the user's aesthetic preference.
  • the cloud trains the relation function, and the photo to be edited can be used as the input, and the edited photo can be used as the output to train the relation function.
  • the cloud can obtain optimized basic image quality algorithm parameters (also called "basic image quality optimization parameters”), such as optimized automatic exposure algorithm parameters, noise reduction algorithm parameters, and beauty algorithm parameters.
  • basic image quality optimization parameters also called "basic image quality optimization parameters”
  • the terminal device uploads the photos before and after editing by the user to the cloud, trains the photo-style optimization network deployed in the cloud, migrates to obtain the photo-style optimization network customized by the user, and synchronously updates the basic image quality algorithm of the terminal device.
  • This kind of private customization method of photo-taking style based on end-cloud collaboration can rely on the strong computing power of the cloud to effectively reduce the hardware requirements of terminal equipment, and use the cloud to model and update basic image quality algorithms to continuously recommend photo-taking styles that are more in line with user habits. Improve the user's photo experience.
  • the mobile phone may not send the complete photo to be edited and the edited photo to the cloud, but only the image features of the photo to be edited and the edited photo to the cloud.
  • the CNN network can be preset in the mobile phone. After the mobile phone obtains the photo to be edited and the edited photo, the mobile phone can perform feature extraction on the photo to be edited and the edited photo through the CNN network, respectively, to obtain the photo to be edited and the edited photo. Image characteristics of the photo. Then, the mobile phone can send the photo to be edited and the image features of the edited photo to the cloud.
  • the cloud can train the aesthetic scoring network according to the image features of the photos to be edited and the edited photos, which will not be repeated here.
  • the mobile phone Compared with the way that the mobile phone directly sends the photo to be edited and the edited photo to the cloud, the mobile phone only sends the image features of the photo to be edited and the edited photo to the cloud, which is used for the cloud to obtain the user-customized photo style optimization network, which can be Better protect user privacy and prevent user privacy leakage.
  • an embodiment of the present application further provides a device-cloud collaboration system.
  • the terminal-cloud collaboration system may include: a terminal device and a cloud.
  • the terminal device is connected to the cloud through a wireless network.
  • the terminal device cooperates with the cloud to implement the photographing method described in the foregoing embodiments.
  • the terminal device has a photographing function.
  • An embodiment of the present application further provides a terminal device, where the terminal device may include a photographing device.
  • the photographing apparatus can be used to implement the functions performed by the terminal device in the photographing method described in the foregoing embodiments.
  • the function of the photographing device can be realized by hardware, or can be realized by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions.
  • FIG. 19 shows a schematic structural diagram of a photographing apparatus provided by an embodiment of the present application.
  • the photographing apparatus may include: a camera module 1901 , a display module 1902 , a sensor module 1903 , a sending module 1904 , a receiving module 1905 and the like.
  • the camera module 1901 is used to obtain the preview image corresponding to the scene to be shot in response to the user operation; the display module 1902 is used to display the preview image obtained by the camera module 1901 on the shooting interface; the sensor module 1903 is used to collect all the images.
  • the scene information of the scene to be shot is described; the sending module 1904 is used to send the preview image obtained by the camera module 1901 and the scene information obtained by the sensor module 1903 to the cloud; the receiving module 1905 is used to receive data from the The composition guide line in the cloud, the composition guide line is used to indicate the composition mode suitable for the scene to be shot; the display module 1902 is also used to display the composition guide line and the preview image on the shooting interface;
  • the camera module 1901 is further configured to obtain a photo corresponding to the scene to be captured in response to the user's photo-taking operation after receiving the user's photo-taking operation.
  • the frequency of sending the scene information to the cloud by the sending module 1904 is less than the frequency of sending the preview image to the cloud.
  • the display module 1902 is further configured to not display the composition auxiliary line in response to the user's operation of turning off the display function of the composition auxiliary line.
  • the sending module 1904 is specifically configured to extract image features of the obtained preview image through a preset convolutional neural network; and send the obtained image features of the preview image to the cloud.
  • the function of extracting image features of the obtained preview image through a preset convolutional neural network can also be accomplished by a separate feature extraction module (not shown in the figure), for example, the device further includes a feature extraction module.
  • the receiving module 1905 is further configured to receive the shooting contour line from the cloud; the display module 1902 is further configured to display the shooting contour line and the composition auxiliary line together with the preview image being displayed on the shooting interface.
  • the sending module 1904 is further configured to send a photo corresponding to the scene to be shot to the cloud; the receiving module 1905 is further configured to receive at least one composition-optimized photo corresponding to the photo from the cloud; the display module 1902, It is also used to display the photo and the at least one composition-optimized photo, and in response to the user's operation of saving the first photo, save the first photo, where the first photo includes the photo and the at least one photo. One or more of the composition-optimized photos.
  • the sending module 1904 is further configured to send the first photo to the cloud, where the first photo is used for the cloud to obtain the user's aesthetic preference.
  • the sending module 1904 is further configured to send the image feature of the first photo to the cloud.
  • Embodiments of the present application further provide a cloud server, where the cloud server can be used to implement the functions performed by the cloud in the photographing methods described in the foregoing embodiments.
  • the function of the photographing device can be realized by hardware, or can be realized by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions.
  • FIG. 20 shows another schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the electronic device may include: a receiving module 2001, a composition module 2002, a sending module 2003, and the like.
  • the receiving module 2001 is used to receive the preview image and scene information corresponding to the scene to be shot from the terminal device;
  • the composition module 2002 is used to generate a composition suitable for the scene to be shot according to the preview image and the scene information
  • the sending module 2003 is configured to send the composition auxiliary line to the terminal device.
  • the composition module 2002 is specifically configured to identify the elements contained in the preview image according to the preview image and scene information, and record the positions and proportions of different elements in the preview image; according to the elements contained in the preview image, As well as the positions and proportions of different elements, a composition guide line corresponding to a composition mode matching the scene to be shot is determined according to a preset matching rule.
  • the matching rules include the correspondence between at least one type of scene to be photographed and the composition auxiliary line, the elements contained in different types of scenes to be photographed, and the positions and proportions of different elements are different.
  • the above-mentioned matching rules may be artificially defined rules.
  • the elements included in the preview image may be sky, sea water, grass, people, and the like.
  • the position of the element refers to the pixel coordinates of the area where the element is located in the preview image
  • the proportion of the element refers to the ratio of the number of pixels in the area where the element is located in the preview image to the number of pixels in the entire preview image.
  • composition module 2002 is specifically configured to use the first method to segment the preview image, and then identify elements contained in the preview image based on the segmentation result and scene information.
  • the scene information is used to assist the composition module 2002 to quickly identify the elements contained in the preview image based on the segmentation result.
  • the location information included in the scene information is seashore, it may indicate that the preview image may contain seawater, which can assist the composition module 2002 to quickly identify whether the preview image contains seawater based on the segmentation result.
  • the first method may be an edge detection-based method, a wavelet transform-based method, a deep learning-based method, or the like. Considering the high performance requirements and low accuracy requirements, the first method can try to use traditional segmentation methods or methods with smaller models.
  • the composition module 2002 can use the deep learning segmentation network U-NET to segment the preview image.
  • the composition module 2002 is specifically used to perform saliency detection on the preview image, and extract the saliency result of the preview image; input the saliency result and scene information of the preview image into the trained artificial intelligence AI network to obtain The probability distribution of the various composition modes corresponding to the preview image output by the AI network; according to the probability distribution of the various composition modes corresponding to the preview image output by the AI network, the composition auxiliary line corresponding to the composition mode matching the scene to be shot is determined.
  • the probability of a composition method that is not suitable for the type of scene to be shot is 0 or close to 0.
  • the composition module 2002 is specifically configured to correct the preview image to obtain a corrected preview image; and according to the corrected preview image and scene information, generate a composition guide line corresponding to a composition mode suitable for the scene to be shot.
  • the cloud includes a line detection network and an image correction module (refer to the aforementioned Figure 2A for details); a composition module 2002 is specifically configured to input the preview image into the line detection network, and detect the elements contained in the preview image through the line detection network. Lines; the image correction module determines the transformation matrix required to correct the preview image according to the lines contained in the preview image, and uses the transformation matrix to correct the preview image to obtain the corrected preview image.
  • the line detection network includes: a backbone network, a connection point prediction module, a line segment sampling module, and a line segment correction module (for details, please refer to the foregoing shown in FIG. 2B ).
  • the composition module 2002 is specifically used to input the preview image into the backbone network, the backbone network extracts the features in the preview image, and outputs the shared convolution feature map corresponding to the preview image to the connection point prediction module; the connection point prediction module outputs according to the shared convolution feature map
  • the candidate connection points corresponding to the preview image are transmitted to the line segment sampling module; the line segment sampling module predicts the lines contained in the preview image according to the candidate connection points.
  • the transformation matrix includes at least a rotation matrix and a homography matrix.
  • the composition module 2002 is further configured to acquire outlines of elements contained in the preview image; and generate a shooting outline suitable for the scene to be shot according to the composition auxiliary lines and the outlines of elements contained in the preview image.
  • the sending module 2003 is further configured to send the shooting contour line to the terminal device.
  • FIG. 21 shows another schematic structural diagram of the electronic device provided by the embodiment of the present application.
  • the electronic device may further include: a composition optimization module 2004 .
  • the receiving module 2001 is further configured to receive a photo corresponding to the scene to be shot from the terminal device.
  • the composition optimization module 2004 is configured to perform composition optimization on the photo to obtain at least one composition-optimized photo corresponding to the photo.
  • the sending module 2003 is further configured to send the at least one composition-optimized photo to the terminal device.
  • the cloud includes a line detection network, an image correction module, a saliency detection module, and an aesthetic scoring network (please refer to the aforementioned Figure 15 for details).
  • the composition optimization module 2004 is specifically configured to input the photo into a line detection network, and detect the lines contained in the photo through the line detection network; determine to correct the photo according to the lines contained in the photo through the image correction module required transformation matrix, and use the transformation matrix to rectify the photo to obtain the rectified photo; perform saliency detection on the rectified photo through the saliency detection module to obtain the saliency of the rectified photo Result: Input the corrected photo and the saliency result into the aesthetic scoring network to obtain a plurality of candidate cropping maps output by the aesthetic scoring network and the corresponding score of each said candidate cropping map; determine a plurality of candidate cropping maps At least one candidate crop with the highest score in the cut is the photo after composition optimization.
  • the initial photos from the terminal device received by the receiving module 2001 are non-portrait photos, such as some landscape/landscape photos.
  • An embodiment of the present application further provides a terminal device, where the terminal device may include a photographing device.
  • the photographing apparatus can be used to implement the functions performed by the terminal device in the photographing method described in the foregoing embodiments.
  • the function of the photographing device can be realized by hardware, or can be realized by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions.
  • FIG. 28 shows a schematic structural diagram of a photographing apparatus provided by an embodiment of the present application.
  • the photographing apparatus may include: a camera module 2801, a sending module 2802, a receiving module 2803, a display module 2804, a processing module 2805, and the like.
  • the camera module 2801 is used to obtain the photo to be edited and the scene information corresponding to the photo to be edited.
  • Sending module 2802 configured to send the photo to be edited and scene information to the cloud, where the photo to be edited and scene information are used by the cloud to optimize the composition of the photo to be edited to obtain the corresponding photo of the photo to be edited of at least one composition-optimized photo.
  • the receiving module 2803 is configured to receive at least one composition-optimized photo from the cloud.
  • the display module 2804 is configured to display the photo to be edited and the at least one photo whose composition has been optimized.
  • the processing module 2805 is configured to select the first photo in response to the user's selection operation on the first photo.
  • the first photo includes the photo to be edited and one or more of the at least one composition-optimized photo.
  • the sending module 2802 is further configured to send the first photo to the cloud, where the first photo is used for the cloud to obtain the user's aesthetic preference.
  • the processing module 2805 is further configured to extract image features of the first photo through a preset convolutional neural network.
  • the sending module 2802 is further configured to send the image feature of the first photo to the cloud.
  • the sending module 2802 is further configured to send the identification information of the first photo to the cloud.
  • Embodiments of the present application further provide a cloud server, where the cloud server can be used to implement the functions performed by the cloud in the photographing methods described in the foregoing embodiments.
  • the function of the photographing device can be realized by hardware, or can be realized by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions.
  • FIG. 29 shows another schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the photographing apparatus may include: a receiving module 2901, a processing module 2902, a sending module 2903, and the like.
  • the receiving module 2901 is used for receiving photos to be edited and scene information from the terminal device.
  • the processing module 2902 is configured to optimize the composition of the photo to be edited according to the photo to be edited and scene information, and obtain at least one photo with optimized composition corresponding to the photo to be edited.
  • the sending module 2903 is configured to send the at least one composition-optimized photo to the terminal device.
  • the receiving module 2901 is further configured to receive a first photo from the terminal device, where the first photo includes the photo and one or more of the at least one composition-optimized photo.
  • the processing module 2902 is further configured to obtain the user's aesthetic preference according to the first photo.
  • the cloud includes a line detection network, an image correction module, a saliency detection module, and an aesthetic scoring network; the processing module 2902 is further configured to input the to-be-edited image into the line detection network, and pass the line detection network Lines contained in the image to be edited are detected. The processing module 2902 is further configured to determine, through the image correction module, a transformation matrix required for correcting the preview image according to the lines contained in the image to be edited, and use the transformation matrix to perform the transformation of the image to be edited. Correction is performed to obtain the corrected image to be edited.
  • the line detection network includes: a backbone network, a connection point prediction module, a line segment sampling module, and a line segment correction module.
  • the line detection network detects the lines contained in the image to be edited, including: after the image to be edited is input into a backbone network, the backbone network extracts features in the image to be edited, and outputs corresponding images of the image to be edited.
  • the shared convolution feature map is sent to the connection point prediction module; the connection point prediction module outputs the candidate connection point corresponding to the image to be edited according to the shared convolution feature map, and transmits it to the line segment sampling module;
  • the line segment sampling module predicts the lines included in the image to be edited according to the candidate connection points.
  • the processing module 2902 is further configured to obtain the first photo corresponding to the identification information stored in the cloud according to the identification information of the first photo, and obtain the aesthetics of the user according to the photo corresponding to the identification information. preference.
  • An embodiment of the present application further provides a terminal device, where the terminal device may include a photographing device.
  • the photographing apparatus can be used to implement the functions performed by the terminal device in the photographing method described in the foregoing embodiments.
  • the function of the photographing device can be realized by hardware, or can be realized by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions.
  • FIG. 30 shows a schematic structural diagram of a photographing apparatus provided by an embodiment of the present application.
  • the photographing apparatus may include: a camera module 3001, a processing module 3002, a sending module 3003, a receiving module 3004, and the like.
  • the camera module 3001 is used to obtain photos to be edited.
  • the processing module 3002 is configured to edit the photo to be edited in response to the user's editing operation, and obtain the edited photo.
  • the sending module 3003 is used to send the photo to be edited and the edited photo to the cloud, and the photo to be edited and the edited photo are used for the cloud to obtain the user's aesthetic preference, and to obtain the basic painting according to the user's aesthetic preference.
  • quality optimization parameters The receiving module 3004 is configured to receive basic image quality optimization parameters from the cloud.
  • the processing module 3002 is further configured to update a local basic image quality algorithm according to the basic image quality optimization parameter.
  • the processing module 3002 is further configured to process the photo to be edited according to a local basic image quality algorithm of the terminal device to obtain an intermediate photo; and to edit the intermediate photo to obtain an edited photo.
  • the processing module 3002 is further configured to extract the image features of the photo to be edited and the image features of the edited photo through a preset convolutional neural network.
  • the sending module 3003 is further configured to send the photo to be edited and the image features of the edited photo to the cloud.
  • Embodiments of the present application further provide a cloud server, which can be used to implement the functions performed by the cloud server in the photographing method described in the foregoing embodiments.
  • the function of the photographing device can be realized by hardware, or can be realized by executing corresponding software by hardware.
  • the hardware or software includes one or more modules or units corresponding to the above functions.
  • FIG. 31 shows another schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the electronic device may include: a receiving module 3101, a processing module 3102, a sending module 3103, and the like.
  • the receiving module 3101 is used to receive the photos to be edited and the edited photos from the terminal device.
  • the processing module 3102 is configured to obtain the user's aesthetic preference according to the photo to be edited and the edited photo, and obtain basic image quality optimization parameters according to the user's aesthetic preference.
  • the sending module 3103 is configured to send the basic image quality optimization parameter to the terminal device.
  • the processing module 3102 is further configured to model the basic image quality algorithm to obtain the relationship function between the parameters of the basic image quality algorithm and the imaging effect.
  • the processing module 3102 is further configured to train the relationship function according to the photo to be edited and the edited photo, so as to obtain the user's aesthetic preference.
  • units units or modules
  • photographing devices in the cloud or photographing devices of terminal equipment is only a division of logical functions, and can be fully or partially integrated into a physical entity in actual implementation. can also be physically separated.
  • all the units in the device can be realized in the form of software calling through the processing element; also can all be realized in the form of hardware; some units can also be realized in the form of software calling through the processing element, and some units can be realized in the form of hardware.
  • each unit can be a separately established processing element, or can be integrated in a certain chip of the device to be implemented, and can also be stored in the memory in the form of a program, which can be called by a certain processing element of the device and execute the unit's processing. Function.
  • all or part of these units can be integrated together, and can also be implemented independently.
  • the processing element described here may also be called a processor, which may be an integrated circuit with signal processing capability.
  • each step of the above method or each of the above units may be implemented by an integrated logic circuit of hardware in the processor element or implemented in the form of software being invoked by the processing element.
  • the units in the above apparatus may be one or more integrated circuits configured to implement the above method, eg, one or more application specific integrated circuits (ASICs), or, one or more A digital signal processor (DSP), or, one or more field programmable gate arrays (FPGA), or a combination of at least two of these integrated circuit forms.
  • ASICs application specific integrated circuits
  • DSP digital signal processor
  • FPGA field programmable gate arrays
  • the processing element can be a general-purpose processor, such as a CPU or other processors that can invoke programs.
  • these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).
  • the unit of the above apparatus for implementing each corresponding step in the above method may be implemented in the form of a processing element scheduler.
  • the apparatus may include a processing element and a storage element, and the processing element invokes a program stored in the storage element to execute the method described in the above method embodiments.
  • the storage element may be a storage element on the same chip as the processing element, ie, an on-chip storage element.
  • the program for performing the above method may be in a storage element on a different chip from the processing element, ie, an off-chip storage element.
  • the processing element calls or loads the program from the off-chip storage element to the on-chip storage element, so as to call and execute the methods described in the above method embodiments.
  • an embodiment of the present application may further provide an apparatus, such as an electronic device, which may include a processor, a memory for storing instructions executable by the processor.
  • an electronic device which may include a processor, a memory for storing instructions executable by the processor.
  • the processor When the processor is configured to execute the above-mentioned instructions, the electronic device implements the steps performed by the terminal device in the photographing method described in the foregoing embodiments.
  • the memory may be located within the electronic device or external to the electronic device.
  • the processor includes one or more.
  • the electronic device can be a mobile terminal such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an AR/VR device, a notebook computer, a UMPC, a netbook, a PDA, etc. Cameras, PTZ cameras, drones and other professional shooting equipment.
  • an embodiment of the present application may further provide an apparatus, such as an electronic device, which may include a processor, a memory for storing instructions executable by the processor.
  • an electronic device which may include a processor, a memory for storing instructions executable by the processor.
  • the processor When the processor is configured to execute the above-mentioned instructions, the electronic device implements the steps performed by the cloud in the photographing method described in the foregoing embodiments.
  • the memory can be located within the electronic device or external to the electronic device.
  • the processor includes one or more.
  • the electronic device may be a computer, a server, or a server cluster composed of multiple servers, or the like.
  • the unit of the apparatus implementing each step in the above method may be configured as one or more processing elements, where the processing elements may be integrated circuits, such as: one or more ASICs, or, one or more Multiple DSPs, or, one or more FPGAs, or a combination of these types of integrated circuits. These integrated circuits can be integrated together to form chips.
  • an embodiment of the present application further provides a chip, which can be applied to an electronic device.
  • the chip includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the processor receives and executes computer instructions from the memory of the electronic device through the interface circuit, so as to realize the above-mentioned embodiments. Steps performed by the terminal device in the photographing method.
  • an embodiment of the present application further provides a chip, which can be applied to an electronic device.
  • the chip includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the processor receives and executes computer instructions from the memory of the electronic device through the interface circuit, so as to realize the above-mentioned embodiments.
  • the steps performed by the cloud in the photographing method are performed by the cloud in the photographing method.
  • the embodiments of the present application further provide a computer program product, including computer-readable codes, when the computer-readable codes are executed in an electronic device, the electronic device enables the electronic device to implement the terminal device in the photographing method described in the foregoing embodiments. steps to perform.
  • the embodiments of the present application further provide a computer program product, including computer-readable codes, when the computer-readable codes are executed in an electronic device, the electronic device enables the cloud-based execution in the photographing method described in the foregoing embodiments.
  • a computer program product including computer-readable codes, when the computer-readable codes are executed in an electronic device, the electronic device enables the cloud-based execution in the photographing method described in the foregoing embodiments.
  • the disclosed apparatus and method may be implemented in other manners.
  • the device embodiments described above are only illustrative.
  • the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components may be Incorporation may either be integrated into another device, or some features may be omitted, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place, or may be distributed to multiple different places . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium.
  • the technical solutions of the embodiments of the present application essentially or contribute to the prior art, or all or part of the technical solutions may be embodied in the form of software products, such as programs.
  • the software product is stored in a program product, such as a computer-readable storage medium, and includes several instructions to cause a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) to execute all of the methods described in the various embodiments of the present application. or part of the steps.
  • the aforementioned storage medium includes: a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, and other media that can store program codes.
  • the embodiments of the present application may further provide a computer-readable storage medium on which computer program instructions are stored.
  • the computer program instructions are executed by the electronic device, the electronic device is made to implement the steps performed by the terminal device in the photographing method described in the foregoing embodiments.
  • the embodiments of the present application may further provide a computer-readable storage medium on which computer program instructions are stored.
  • the computer program instructions are executed by the electronic device, the electronic device is made to implement the steps executed in the cloud in the photographing method described in the foregoing embodiments.
  • all the functions performed by the cloud described in the foregoing embodiments may also be fully integrated in the terminal device.
  • an embodiment of the present application may further provide a photographing method, and the photographing method may be applied to a terminal device.
  • the method may include: after the terminal device starts and runs the photographing application program, obtains a preview image corresponding to the scene to be photographed, and displays it on the photographing interface; at the same time, the terminal device collects scene information of the scene to be photographed through a sensor.
  • the terminal device generates, according to the preview image and scene information, a composition auxiliary line corresponding to a composition mode suitable for the scene to be shot.
  • the terminal device displays the composition guide line on the shooting interface together with the preview image being displayed.
  • the terminal device After receiving the user's photographing operation, the terminal device obtains a photograph corresponding to the scene to be photographed in response to the user's photographing operation.
  • the photographing method can also guide the user in composition through the composition auxiliary line when the user uses the terminal device to take a photo.
  • users can more easily know which composition method to use according to the composition guideline recommended by the cloud. They do not need to judge the composition method by themselves, and do not need complex composition operations.
  • the composition guideline can indicate the composition method suitable for the scene to be shot for the user, and guide the user to adjust the shooting position, angle, etc. of the terminal device according to the composition method indicated by the composition guideline to complete the composition.
  • the user can also complete a better composition when taking a photo, and the user experience can be better.
  • the preview image may be corrected first to obtain the corrected preview image. Then, the terminal device may specifically generate a composition auxiliary line corresponding to a composition mode suitable for the scene to be shot according to the corrected preview image and scene information.
  • the terminal device may also acquire contour lines of elements contained in the preview image; and generate a shooting contour line suitable for the scene to be shot according to the composition auxiliary lines and the contour lines of elements contained in the preview image. Then, the terminal device may display the shooting outline and composition auxiliary line on the shooting interface together with the preview image being displayed.
  • the terminal device can further optimize the composition of the photograph locally, and display the photograph and the composition. Optimized photo.
  • the basic principle that the terminal device generates the composition guide line corresponding to the composition mode suitable for the scene to be shot according to the preview image and the scene information is the same as that in the foregoing embodiment, the cloud generates the composition corresponding to the composition mode suitable for the scene to be shot according to the preview image and scene information.
  • the basic principle of auxiliary lines is the same.
  • the principle of generating the shooting contour line by the terminal device is the same as that of the cloud generating the shooting contour line.
  • the principle of composition optimization performed by the terminal device is the same as the principle of composition optimization performed by the cloud. I won't go into details here.

Abstract

本申请提供一种拍照方法及装置，涉及拍照领域。该方法包括：终端设备启动运行拍照应用程序后，获取待拍摄场景对应的预览图像，并在拍摄界面进行显示；同时，终端设备通过传感器采集待拍摄场景的场景信息。终端设备向云端发送获取到的预览图像和场景信息。云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线，并将构图辅助线发送给终端设备。终端设备接收到构图辅助线后，将构图辅助线与正在显示的预览图像一同显示在拍摄界面。终端设备接收到用户的拍照操作后，响应于用户的拍照操作，获取待拍摄场景对应的照片。用户使用终端设备进行拍照时，构图辅助线可以引导用户按照构图辅助线指示的构图方式完成构图。

Description

拍照方法及装置
本申请要求下列优先权：2021年4月26日提交中国专利局、申请号为202110454641.4、发明名称为“拍照方法及装置”的中国专利申请的优先权；2021年5月14日提交中国专利局、申请号为202110524940.0、发明名称为“图像处理方法及相关设备”的中国专利申请的优先权；2021年5月14日提交中国专利局、申请号为202110524967.X、发明名称为“拍照方法及相关设备”的中国专利申请的优先权；2021年8月5日提交中国专利局、申请号为202110897914.2、申请名称为“拍照方法及装置”的中国专利申请的优先权，上述申请的全部内容引用结合在本申请中。
技术领域
本申请实施例涉及拍照领域，尤其涉及一种拍照方法及装置。
背景技术
随着手机拍照技术的飞速发展，使用手机拍照已经成为了人类生活中不可或缺的一部分。用户在使用手机进行拍照时，对构图方式的调整会影响到最终拍摄到的照片质量。例如，好的构图可以让拍出的照片更加出色，而错误的构图会导致拍摄出的照片达不到预期效果。更甚至，在拍照时进行较好的构图往往可以化腐朽为神奇，弥补用户拍照经验不足的缺点。
目前，手机在相机启动运行时，相机的拍摄界面中可以显示参考线，如：九宫格网格线。用户使用手机进行拍照时，可以参考拍摄界面中显示的九宫格网格线对手机的拍摄位置、角度等进行调整，从而完成三分法构图、对角线构图、对称构图等多种不同方式的构图。
但是，上述通过在拍摄界面中显示参考线辅助用户完成构图的方式，参考线的引导作用有限，最终构图好坏与用户的拍摄经验和手法强相关，一些没有丰富拍摄经验的用户可能并不懂得如何根据参考线对手机的拍摄位置、角度等进行调整以完成不同方式的构图。
发明内容
本申请实施例提供一种拍照方法及装置，可以在用户使用终端设备进行拍照时，结合当前的拍摄场景在终端设备提供的拍摄界面中显示构图辅助线，对用户进行构图引导。
第一方面，本申请实施例提供一种拍照方法，所述方法应用于端云协同系统。端云协同系统包括终端设备和云端。终端设备通过无线网络与云端连接。所述方法包括：
终端设备响应于用户的操作，获取并显示待拍摄场景对应的预览图像；终端设备采集待拍摄场景的场景信息。终端设备向云端发送获取到的预览图像和场景信息。云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线，并将构图辅助线发送给终端设备。终端设备接收到构图辅助线后，将构图辅助线与所述 预览图像显示在拍摄界面。终端设备响应于用户的拍照操作，获取待拍摄场景对应的照片。
其中，构图辅助线也可以被称为参考线、构图参考线、构图线等，在此对构图辅助线的名称并不作限制。
该拍照方法能够在用户使用终端设备进行拍照时，结合当前的拍摄场景在终端设备提供的拍摄界面中显示构图辅助线，对用户进行构图引导。其中，构图辅助线可以为用户指示适合待拍摄场景的构图方式，引导用户按照构图辅助线指示的构图方式对终端设备的拍摄位置、角度等进行调整以完成构图，从而可以使得没有丰富拍摄经验的用户在进行拍照时也能够完成较好的构图，用户的体验可以更好。
也即，该拍照方法可以使得用户在使用终端设备进行拍照时，能够根据云端推荐的构图辅助线，更轻松地知道通过哪种构图方式进行构图，不需要自己判断构图方式，也不要复杂的构图操作。
另外，云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线时，场景信息可以辅助云端快速识别预览图像中包含的元素，提高云端生成构图辅助线的效率，降低延时。
示例性地，传感器至少包括位置传感器、气压传感器，温度传感器，环境光传感器。场景信息至少包括待拍摄场景对应的位置信息、气压信息、温度信息、光强信息。
可选地，终端设备可以按照第一帧率获取待拍摄场景对应的预览图像，并在拍摄界面进行显示。同时，终端设备按照第一频率，通过传感器采集待拍摄场景的场景信息。
在一种可能的设计中，终端设备可以按照第二频率向云端发送获取到的预览图像和场景信息。其中，第二频率的数值(或称为大小)小于或等于第一帧率的数值和第一频率的数值中的最小值。第一帧率和第一频率的大小可以相同或不同。
在另外一种可能的设计中，终端设备向云端发送场景信息的频率，小于或等于终端设备向云端发送预览图像的频率。例如，终端设备按照第三频率向云端发送获取到的预览图像、按照第四频率向云端发送获取到的场景信息，第三频率大于第四频率。
一种实施方式中，所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线，包括：云端根据预览图像和场景信息，识别预览图像中包含的元素，并记录预览图像中不同元素的位置和所占比例。云端根据预览图像中包含的元素，以及不同元素的位置和所占比例，按照预设的匹配规则确定与待拍摄场景匹配的构图方式对应的构图辅助线。
其中，匹配规则包括至少一种类型的待拍摄场景与构图辅助线之间的对应关系，不同类型的待拍摄场景中包含的元素、以及不同元素的位置和所占比例不同。
上述匹配规则可以是人为定义的规则。
示例性地，预览图像中包含的元素可以是天空、海水、草地、人物等。元素的位置是指预览图像中元素所在区域的像素点坐标，元素所占的比例是指预览图像中元素所在区域的像素点数量占整个预览图像的像素点数量的比值。
可选地，所述云端根据预览图像和场景信息，识别预览图像中包含的元素，包括： 云端采用第一方法对预览图像进行分割，然后基于分割结果、结合场景信息识别预览图像中包含的元素。
其中，场景信息用于辅助云端基于分割结果快速识别预览图像中包含的元素。
例如，当场景信息包括的位置信息为海边时，可以表示预览图像中可能含有海水，能够辅助云端基于分割结果快速识别出预览图像中是否包含海水。
示例性地，第一方法可以是基于边缘检测的方法、基于小波变换的方法、基于深度学习的方法等。考虑到性能要求高，精度要求较低，第一方法可以尽量选用传统分割方法或者模型较小的方法。例如，云端可以采用深度学习分割网络U-NET对预览图像进行分割。
另一种实施方式中，所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线，包括：云端对预览图像进行显著性检测，提取预览图像的显著性结果；云端将预览图像的显著性结果和场景信息输入训练好的人工智能AI网络，得到AI网络输出的预览图像对应的多种构图方式的概率分布；云端根据AI网络输出的预览图像对应的多种构图方式的概率分布，确定与待拍摄场景匹配的构图方式对应的构图辅助线。
对某种类型的待拍摄场景而言，AI网络输出的概率分布中，不适合该类型的待拍摄场景的构图方式的概率为0或接近于0。
一些实施例中，云端向终端设备返回的构图辅助线可以为预览图像中的多个像素点坐标组成的坐标数据集合。
另外一些实施例中，云端也可以向终端设备返回一张包含构图辅助线的图像，该图像中，构图辅助线所在区域的像素点的像素值可以为0，除构图辅助线之外的区域的像素点的像素值可以为P，P为大于0、小于或等于255的整数。可选地，除构图辅助线之外的区域的像素点的像素值可以均为255。
一些实施例中，所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线之前，所述方法还包括：云端对预览图像进行矫正，得到矫正后的预览图像。所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线，包括：云端根据矫正后的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线。
示例性地，云端包括线条检测网络和图像矫正模块；所述云端对预览图像进行矫正，得到矫正后的预览图像，包括：云端将预览图像输入线条检测网络，通过线条检测网络检测预览图像中包含的线条；云端通过图像矫正模块根据预览图像中包含的线条，确定对预览图像进行矫正所需的变换矩阵，并采用变换矩阵对预览图像进行矫正，得到矫正后的预览图像。
可选地，线条检测网络包括：主干网络、连接点预测模块、线段采样模块、以及线段校正模块；所述线条检测网络检测预览图像中包含的线条，包括：将预览图像输入主干网络后，主干网络提取预览图像中的特征，输出预览图像对应的共享卷积特征图至连接点预测模块；连接点预测模块根据共享卷积特征图输出预览图像对应的候选连接点，并传输给线段采样模块；线段采样模块根据候选连接点预测出预览图像包含 的线条。
可选地，变换矩阵至少包括旋转矩阵、单应性矩阵。
云端对预览图像进行矫正，可以使得云端根据矫正后的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线时，能够更加准确地识别到预览图像中包含的元素、以及不同元素的位置和所占比例，进而使生成的辅助线更加符合待拍摄场景对应的构图方式。
可选地，所述方法还包括：终端设备响应于用户关闭构图辅助线的显示功能的操作，不显示构图辅助线。
对于一些具有丰富拍摄经验的用户而言，可能更期待在拍照时按照自己的意愿去构图，不希望受到构图辅助线的干扰，那么通过这种方式可以进一步考虑更多用户的需求，提升用户的体验。
可选地，所述终端设备向云端发送获取到的预览图像，包括：终端设备通过预设的卷积神经网络提取获取到的预览图像的图像特征；终端设备向云端发送获取到的预览图像的图像特征。
相对于直接向云端发送预览图像的方式而言，终端设备只向云端发送预览图像的图像特征，用于云端生成适合待拍摄场景的构图方式对应的构图辅助线，可以更好的保护用户隐私，防止用户隐私泄露。
可选地，所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线之后，所述方法还包括：云端获取预览图像中包含的元素的轮廓线；云端根据构图辅助线、以及预览图像中包含的元素的轮廓线，生成适合待拍摄场景的拍摄轮廓线，并将拍摄轮廓线发送给终端设备；终端设备接收到拍摄轮廓线后，将拍摄轮廓线、构图辅助线与正在显示的预览图像一同显示在拍摄界面。
与上述构图辅助线的实现方式类似，云端向终端设备返回的拍摄轮廓线也可以是预览图像中的多个像素点坐标组成的坐标数据集合(与构图辅助线的坐标数据集合不同)。或者，云端也可以向终端设备返回一张包含构图辅助线和拍摄轮廓线的图像，该图像中，构图辅助线所在区域的像素点的像素值可以为0；拍摄轮廓线所在区域的像素点的像素值也可以为0。除构图辅助线和拍摄轮廓线之外的区域的像素点的像素值均可以为P，P为大于0、小于或等于255的整数。又或者，构图辅助线所在区域的像素点的像素值与拍摄轮廓线所在区域的像素点的像素值可以不同。
用户使用终端设备进行拍照前，参考拍摄界面中显示的构图辅助线，对终端设备的拍摄位置、角度等进行调整时，还可以进一步参考拍摄界面显示的拍摄轮廓线，将预览画面中的元素移动至拍摄轮廓线所指示的位置。通过这种方式，可以更进一步地简化用户进行构图的操作，提升用户体验。
可选地，所述方法还包括：终端设备向云端发送待拍摄场景对应的照片；云端对所述照片进行构图优化，得到所述照片对应的至少一张构图优化后的照片，并将所述至少一张构图优化后的照片发送给终端设备；终端设备显示所述照片、以及所述至少一张构图优化后的照片；终端设备响应于用户对第一照片的保存操作，保存第一照片，第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张。
云端通过对终端设备拍摄的照片(可称为初始照片)进行进一步的构图优化，并返回给终端设备构图优化后的照片，可以提高构图成功率，构图优化后的照片不会出现旋转、透视等畸变，能够为用户提供更好的照片选择。另外，终端设备同时显示用户拍摄的初始照片和构图优化后的照片，可以考虑到用户的需求，使得用户既可以选择自己拍摄的照片，也可以选择云端推荐的构图优化后的照片。
示例性地，云端包括线条检测网络、图像矫正模块、显著性检测模块、以及美学评分网络。所述云端对所述照片进行构图优化，得到所述照片对应的至少一张构图优化后的照片，包括：云端将所述照片输入线条检测网络，通过线条检测网络检测所述照片中包含的线条；云端通过图像矫正模块根据所述照片中包含的线条，确定对所述照片进行矫正所需的变换矩阵，并采用变换矩阵对所述照片进行矫正，得到矫正后的照片；云端通过显著性检测模块对所述矫正后的照片进行显著性检测，得到所述矫正后的照片的显著性结果；云端将所述矫正后的照片、以及显著性结果输入美学评分网络，得到美学评分网络输出的多张候选裁切图、以及每张所述候选裁切图对应的评分；云端确定多张候选裁切图中评分最高的至少一张候选裁切图为构图优化后的照片。
一些实施例中，终端设备在向云端发送待拍摄场景对应的初始照片之前，可以先检测初始照片是否为人像照片，如果初始照片是人像照片，则终端设备可以直接保存人像照片，不向云端发送初始照片，以更好的保护用户隐私，防止用户隐私泄露。
可选地，所述方法还包括：终端设备向云端发送所述第一照片；云端根据第一照片，获取用户的美学偏好。
在一些实施例中，终端设备向所述云端发送所述第一照片，包括：终端设备通过预设的卷积神经网络提取第一照片的图像特征；终端设备向云端发送第一照片的图像特征。
在一些实施例中，终端设备向所述云端发送所述第一照片，包括：终端设备向所述云端发送所述第一照片的标识信息。云端根据所述第一照片获取用户的美学偏好，包括：云端根据所述第一照片的标识信息获取存储在云端的第一照片，并根据所述第一照片获取用户的美学偏好。
一些实施例中，云端根据所述第一照片获取用户的美学偏好，包括：云端根据所述第一照片获取对应的场景信息，并根据所述第一照片和场景信息获取用户的美学偏好。
终端设备将用户的拍照构图选择结果上传至云端，云端对构图优化网络进行迁移训练，得到用户私人定制的构图优化网络。这种端云协同的拍照构图私人定制方法，可以依靠云端的强算力，有效降低终端设备的硬件要求，持续推荐更加符合用户习惯的拍照构图，减少用户学习成本，提升用户拍照体验。
第二方面，本申请实施例提供一种端云协同系统，包括终端设备和云端；终端设备通过无线网络与云端连接。终端设备与云端配合实现如第一方面及第一方面中的任意一种实现方式中所述的方法。
上述第二方面所具备的有益效果，可参考第一方面中所述，在此不再赘述。
第三方面，本申请实施例提供一种拍照方法，所述方法应用于终端设备，终端设 备通过无线网络与云端连接。所述方法包括：
终端设备响应于用户的操作，获取并显示待拍摄场景对应的预览图像；同时，终端设备采集待拍摄场景的场景信息。终端设备向云端发送获取到的预览图像和场景信息。终端设备接收来自云端的构图辅助线，构图辅助线用于指示适合待拍摄场景的构图方式。终端设备将构图辅助线与正在显示的预览图像显示在拍摄界面。终端设备响应于用户的拍照操作，获取待拍摄场景对应的照片。
示例性地，当用户需要使用终端设备进行拍照时，可以先启动终端设备的拍照应用程序。例如，用户可以点击或触摸终端设备上的相机的图标，终端设备可以响应于用户对相机的图标的点击或触摸操作，启动运行相机(或者，用户还可以通过语音助手启动相机，不作限制)。
示例性地，终端设备通过拍照应用程序提供的拍摄界面还可以包括一个拍照按键，该拍照按键的实质可以为拍摄界面中显示的一个功能控件。用户在完成构图之后，可以点击或触摸拍照按键，终端设备可以响应于用户对拍照按键的点击或触摸操作进行拍照，从而获取到待拍摄场景对应的照片。
示例性地，传感器至少包括位置传感器、气压传感器，温度传感器，环境光传感器。场景信息至少包括待拍摄场景对应的位置信息、气压信息、温度信息、光强信息。
可选地，终端设备可以按照第一帧率获取待拍摄场景对应的预览图像，并在拍摄界面进行显示。同时，终端设备按照第一频率，通过传感器采集待拍摄场景的场景信息。
在一种可能的设计中，终端设备可以按照第二频率向云端发送获取到的预览图像和场景信息。其中，第二频率的数值(或称为大小)小于或等于第一帧率的数值和第一频率的数值中的最小值。第一帧率和第一频率的大小可以相同或不同。
在另外一种可能的设计中，终端设备向云端发送场景信息的频率，小于或等于终端设备向云端发送预览图像的频率。例如，终端设备按照第三频率(30次/秒)向云端发送获取到的预览图像、按照第四频率(10次/秒)向云端发送获取到的场景信息，第三频率大于第四频率。在用户使用终端设备进行拍照时，终端设备在拍摄界面中显示构图辅助线，可以对用户进行构图引导。具体地，构图辅助线可以为用户指示适合待拍摄场景的构图方式，引导用户按照构图辅助线指示的构图方式对终端设备的拍摄位置、角度等进行调整以完成构图，从而可以使得没有丰富拍摄经验的用户在进行拍照时也能够完成较好的构图，用户的体验可以更好。用户基于构图辅助线可以更轻松地知道通过哪种构图方式进行构图，不需要自己判断构图方式，也不要复杂的构图操作。
可选地，终端设备将构图辅助线与正在显示的预览图像一同显示在拍摄界面的同时，还可以在拍摄界面中显示文字提示，提示用户按照构图辅助线引导的方式完成构图。
示例性地，该文字提示也可以是由云端在生成构图辅助线时，一并生成的。云端可以将构图辅助线和文字提示一并发送给终端设备。待拍摄场景不同或者构图辅助线不同时，文字提示可能不同。
可选地，终端设备没有接收到来自云端的构图辅助线时，可以不显示构图辅助线。 或者，终端设备可以显示普通的九宫格辅助线。
可选地，所述方法还包括：终端设备响应于用户关闭构图辅助线的显示功能的操作，不显示构图辅助线。
示例性地，终端设备可以为用户提供用于关闭构图辅助线的显示功能的功能控件或物理按键，用户关闭构图辅助线的显示功能的操作可以是指用户点击或触摸前述功能控件或物理按键的操作。终端设备可以响应于用户点击或触摸前述功能控件或物理按键的操作，不显示构图辅助线。
对于一些具有丰富拍摄经验的用户而言，可能更期待在拍照时按照自己的意愿去构图，不希望受到构图辅助线的干扰。终端设备响应于用户关闭构图辅助线的显示功能的操作，不显示构图辅助线，可以进一步考虑更多用户的需求，提升用户的体验。
可选地，终端设备可以只在构图辅助线显示功能开启时，将预览图像发送到云端以获取云端根据预览图像返回的构图辅助线。当关闭构图辅助线显示之后，终端设备在每次进行拍照时，拍摄界面中都不会显示构图辅助线。直至用户再次开启构图辅助线显示时，终端设备才会在拍摄界面中显示构图辅助线。
可选地，所述终端设备向云端发送获取到的预览图像，包括：终端设备通过预设的卷积神经网络提取获取到的预览图像的图像特征；终端设备向云端发送获取到的预览图像的图像特征。
相对于直接向云端发送预览图像的方式而言，终端设备只向云端发送预览图像的图像特征，用于云端生成适合待拍摄场景的构图方式对应的构图辅助线，可以更好的保护用户隐私，防止用户隐私泄露。
可选地，所述方法还包括：终端设备接收来自云端的拍摄轮廓线；终端设备将拍摄轮廓线、构图辅助线与正在显示的预览图像一同显示在拍摄界面。
用户使用终端设备进行拍照前，参考拍摄界面中显示的构图辅助线，对终端设备的拍摄位置、角度等进行调整时，还可以进一步参考拍摄界面显示的拍摄轮廓线，将预览画面中的元素移动至拍摄轮廓线所指示的位置。通过这种方式，可以更进一步地简化用户进行构图的操作，提升用户体验。
可选地，所述方法还包括：终端设备向云端发送待拍摄场景对应的照片；终端设备接收来自云端的所述照片对应的至少一张构图优化后的照片；终端设备显示所述照片、以及所述至少一张构图优化后的照片；终端设备响应于用户对第一照片的保存操作，保存第一照片，第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张。
其中，终端设备显示的构图优化后的照片，构图成功率更高。构图优化后的照片不会出现旋转、透视等畸变，能够为用户提供更好的照片选择。
终端设备同时显示用户拍摄的初始照片和构图优化后的照片，可以考虑到用户的需求，使得用户既可以选择自己拍摄的照片，也可以选择云端推荐的构图优化后的照片。
一些实施例中，终端设备在向云端发送待拍摄场景对应的初始照片之前，可以先检测初始照片是否为人像照片，如果初始照片是人像照片，则终端设备可以直接保存 人像照片，不向云端发送初始照片，以更好的保护用户隐私，防止用户隐私泄露。
第四方面，本申请实施例提供一种终端设备，终端设备可以包括拍照装置，该装置可以用于实现上述第三方面所述的方法。该装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元，例如，相机模块，显示模块，传感器模块，发送模块，接收模块等。
其中，相机模块，用于启动运行拍照应用程序后，获取待拍摄场景对应的预览图像；显示模块，用于在拍摄界面显示所述相机模块获取的预览图像；传感器模块，用于采集所述待拍摄场景的场景信息；发送模块，用于向云端发送所述相机模块获取到的预览图像、以及所述传感器模块获取到的场景信息；接收模块，用于接收来自所述云端的构图辅助线，所述构图辅助线用于指示适合所述待拍摄场景的构图方式；所述显示模块，还用于将所述构图辅助线与正在显示的预览图像一同显示在所述拍摄界面；所述相机模块，还用于接收到用户的拍照操作后，响应于用户的拍照操作，获取所述待拍摄场景对应的照片。
可选地，发送模块向云端发送场景信息的频率小于向云端发送预览图像的频率。
可选地，显示模块，还用于响应于用户关闭构图辅助线的显示功能的操作，不显示构图辅助线。
可选地，发送模块，具体用于通过预设的卷积神经网络提取获取到的预览图像的图像特征；向云端发送获取到的预览图像的图像特征。或者，通过预设的卷积神经网络提取获取到的预览图像的图像特征的功能，也可以由单独的一个特征提取模块完成，如：该装置还包括特征提取模块。
可选地，接收模块，还用于接收来自云端的拍摄轮廓线；显示模块，还用于将拍摄轮廓线、构图辅助线与正在显示的预览图像一同显示在拍摄界面。
可选地，发送模块，还用于向云端发送待拍摄场景对应的照片；接收模块，还用于接收来自云端的所述照片对应的至少一张构图优化后的照片；显示模块，还用于显示所述照片、以及所述至少一张构图优化后的照片，并响应于用户对第一照片的保存操作，保存第一照片，第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张。
第五方面，本申请实施例提供一种电子设备，包括：处理器，用于存储所述处理器可执行指令的存储器；所述处理器被配置为执行所述指令时，使得所述电子设备实现如第三方面及第三方面中的任意一种实现方式中所述的方法。
该电子设备可以是手机、平板电脑、可穿戴设备、车载设备、AR/VR设备、笔记本电脑、超级移动个人计算机、上网本、个人数字助理等移动终端，或者，也可以是数码相机、单反相机/微单相机、运动摄像机、云台相机、无人机等专业的拍摄设备。
第六方面，本申请实施例提供一种计算机可读存储介质，其上存储有计算机程序指令；当计算机程序指令被电子设备执行时，使得电子设备实现如第三方面及第三方面中的任意一种实现方式中所述的方法。
第七方面，本申请实施例还提供一种计算机程序产品，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，使得电子设备实现前述第三方面及第三方 面中的任意一种实现方式中所述的方法。
上述第四方面至第七方面所具备的有益效果，可参考第三方面及第三方面中的任意一种实现方式中所述，在此不再赘述。
第八方面，本申请实施例还提供一种拍照方法，所述方法应用于云端，云端通过无线网络与终端设备连接；所述方法包括：
云端接收来自终端设备的待拍摄场景对应的预览图像和场景信息。云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线。云端向终端设备发送构图辅助线。
其中，构图辅助线也可以被称为参考线、构图参考线、构图线等，在此对构图辅助线的名称并不作限制。
云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线后，向终端设备发送构图辅助线，可以使得用户使用终端设备进行拍照时，构图辅助线可以对用户进行构图引导，为用户指示适合待拍摄场景的构图方式，引导用户按照构图辅助线指示的构图方式对终端设备的拍摄位置、角度等进行调整以完成构图。从而可以使得没有丰富拍摄经验的用户在进行拍照时也能够完成较好的构图，用户的体验可以更好。
另外，云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线时，场景信息可以辅助云端快速识别预览图像中包含的元素，提高云端生成构图辅助线的效率，降低延时。
一种实施方式中，所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线，包括：云端根据预览图像和场景信息，识别预览图像中包含的元素，并记录预览图像中不同元素的位置和所占比例。云端根据预览图像中包含的元素，以及不同元素的位置和所占比例，按照预设的匹配规则确定与待拍摄场景匹配的构图方式对应的构图辅助线。
其中，匹配规则包括至少一种类型的待拍摄场景与构图辅助线之间的对应关系，不同类型的待拍摄场景中包含的元素、以及不同元素的位置和所占比例不同。
上述匹配规则可以是人为定义的规则。
示例性地，预览图像中包含的元素可以是天空、海水、草地、人物等。元素的位置是指预览图像中元素所在区域的像素点坐标，元素所占的比例是指预览图像中元素所在区域的像素点数量占整个预览图像的像素点数量的比值。
可选地，所述云端根据预览图像和场景信息，识别预览图像中包含的元素，包括：云端采用第一方法对预览图像进行分割，然后基于分割结果、结合场景信息识别预览图像中包含的元素。
其中，场景信息用于辅助云端基于分割结果快速识别预览图像中包含的元素。
例如，当场景信息包括的位置信息为海边时，可以表示预览图像中可能含有海水，能够辅助云端基于分割结果快速识别出预览图像中是否包含海水。
示例性地，第一方法可以是基于边缘检测的方法、基于小波变换的方法、基于深度学习的方法等。考虑到性能要求高，精度要求较低，第一方法可以尽量选用传统分 割方法或者模型较小的方法。例如，云端可以采用深度学习分割网络U-NET对预览图像进行分割。
另一种实施方式中，所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线，包括：云端对预览图像进行显著性检测，提取预览图像的显著性结果；云端将预览图像的显著性结果和场景信息输入训练好的人工智能AI网络，得到AI网络输出的预览图像对应的多种构图方式的概率分布；云端根据AI网络输出的预览图像对应的多种构图方式的概率分布，确定与待拍摄场景匹配的构图方式对应的构图辅助线。
对某种类型的待拍摄场景而言，AI网络输出的概率分布中，不适合该类型的待拍摄场景的构图方式的概率为0或接近于0。
一些实施例中，所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线之前，所述方法还包括：云端对预览图像进行矫正，得到矫正后的预览图像。所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线，包括：云端根据矫正后的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线。
示例性地，云端包括线条检测网络和图像矫正模块；所述云端对预览图像进行矫正，得到矫正后的预览图像，包括：云端将预览图像输入线条检测网络，通过线条检测网络检测预览图像中包含的线条；云端通过图像矫正模块根据预览图像中包含的线条，确定对预览图像进行矫正所需的变换矩阵，并采用变换矩阵对预览图像进行矫正，得到矫正后的预览图像。
可选地，线条检测网络包括：主干网络、连接点预测模块、线段采样模块、以及线段校正模块；所述线条检测网络检测预览图像中包含的线条，包括：将预览图像输入主干网络后，主干网络提取预览图像中的特征，输出预览图像对应的共享卷积特征图至连接点预测模块；连接点预测模块根据共享卷积特征图输出预览图像对应的候选连接点，并传输给线段采样模块；线段采样模块根据候选连接点预测出预览图像包含的线条。
可选地，变换矩阵至少包括旋转矩阵、单应性矩阵。
云端对预览图像进行矫正，可以使得云端根据矫正后的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线时，能够更加准确地识别到预览图像中包含的元素、以及不同元素的位置和所占比例，进而使生成的辅助线更加符合待拍摄场景对应的构图方式。
可选地，所述云端根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线之后，所述方法还包括：云端获取预览图像中包含的元素的轮廓线；云端根据构图辅助线、以及预览图像中包含的元素的轮廓线，生成适合待拍摄场景的拍摄轮廓线。云端向终端设备发送拍摄轮廓线。
云端向终端设备发送的拍摄轮廓线，可以使得用户使用终端设备进行拍照时，参考拍摄界面中显示的构图辅助线，对终端设备的拍摄位置、角度等进行调整的同时，还可以进一步参考拍摄界面显示的拍摄轮廓线，将预览画面中的元素移动至拍摄轮廓 线所指示的位置。从而更进一步地简化用户进行构图的操作，提升用户体验。
可选地，所述方法还包括：云端接收来自终端设备的待拍摄场景对应的照片。云端对所述照片进行构图优化，得到所述照片对应的至少一张构图优化后的照片。云端向终端设备发送所述至少一张构图优化后的照片。
云端通过对终端设备拍摄的照片(可称为初始照片)进行进一步的构图优化，并返回给终端设备构图优化后的照片，可以提高构图成功率，构图优化后的照片不会出现旋转、透视等畸变，能够为用户提供更好的照片选择。
示例性地，云端包括线条检测网络、图像矫正模块、显著性检测模块、以及美学评分网络。所述云端对所述照片进行构图优化，得到所述照片对应的至少一张构图优化后的照片，包括：云端将所述照片输入线条检测网络，通过线条检测网络检测所述照片中包含的线条；云端通过图像矫正模块根据所述照片中包含的线条，确定对所述照片进行矫正所需的变换矩阵，并采用变换矩阵对所述照片进行矫正，得到矫正后的照片；云端通过显著性检测模块对所述矫正后的照片进行显著性检测，得到所述矫正后的照片的显著性结果；云端将所述矫正后的照片、以及显著性结果输入美学评分网络，得到美学评分网络输出的多张候选裁切图、以及每张所述候选裁切图对应的评分；云端确定多张候选裁切图中评分最高的至少一张候选裁切图为构图优化后的照片。
一些实施例中，云端接收到的来自终端设备的初始照片为非人像照片，如：可以是一些风景/风光照片。
第九方面，本申请实施例提供一种云端服务器，该装置可以用于实现上述第八方面所述的方法。该装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元，例如，接收模块，构图模块，发送模块等。
其中，接收模块，用于接收来自终端设备的待拍摄场景对应的预览图像和场景信息；构图模块，用于根据所述预览图像和所述场景信息，生成适合所述待拍摄场景的构图方式对应的构图辅助线；发送模块，用于向所述终端设备发送所述构图辅助线。
一种实施方式中，构图模块，具体用于根据预览图像和场景信息，识别预览图像中包含的元素，并记录预览图像中不同元素的位置和所占比例；根据预览图像中包含的元素，以及不同元素的位置和所占比例，按照预设的匹配规则确定与待拍摄场景匹配的构图方式对应的构图辅助线。
其中，匹配规则包括至少一种类型的待拍摄场景与构图辅助线之间的对应关系，不同类型的待拍摄场景中包含的元素、以及不同元素的位置和所占比例不同。
上述匹配规则可以是人为定义的规则。
示例性地，预览图像中包含的元素可以是天空、海水、草地、人物等。元素的位置是指预览图像中元素所在区域的像素点坐标，元素所占的比例是指预览图像中元素所在区域的像素点数量占整个预览图像的像素点数量的比值。
可选地，构图模块，具体用于采用第一方法对预览图像进行分割，然后基于分割结果、结合场景信息识别预览图像中包含的元素。
其中，场景信息用于辅助构图模块基于分割结果快速识别预览图像中包含的元素。
例如，当场景信息包括的位置信息为海边时，可以表示预览图像中可能含有海水，能够辅助构图模块基于分割结果快速识别出预览图像中是否包含海水。
示例性地，第一方法可以是基于边缘检测的方法、基于小波变换的方法、基于深度学习的方法等。考虑到性能要求高，精度要求较低，第一方法可以尽量选用传统分割方法或者模型较小的方法。例如，构图模块可以采用深度学习分割网络U-NET对预览图像进行分割。
另一种实施方式中，构图模块，具体用于对预览图像进行显著性检测，提取预览图像的显著性结果；将预览图像的显著性结果和场景信息输入训练好的人工智能AI网络，得到AI网络输出的预览图像对应的多种构图方式的概率分布；根据AI网络输出的预览图像对应的多种构图方式的概率分布，确定与待拍摄场景匹配的构图方式对应的构图辅助线。
对某种类型的待拍摄场景而言，AI网络输出的概率分布中，不适合该类型的待拍摄场景的构图方式的概率为0或接近于0。
一些实施例中，构图模块，具体用于对预览图像进行矫正，得到矫正后的预览图像；根据矫正后的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线。
示例性地，云端包括线条检测网络和图像矫正模块；构图模块，具体用于将预览图像输入线条检测网络，通过线条检测网络检测预览图像中包含的线条；通过图像矫正模块根据预览图像中包含的线条，确定对预览图像进行矫正所需的变换矩阵，并采用变换矩阵对预览图像进行矫正，得到矫正后的预览图像。
可选地，线条检测网络包括：主干网络、连接点预测模块、线段采样模块、以及线段校正模块；构图模块，具体用于将预览图像输入主干网络，主干网络提取预览图像中的特征，输出预览图像对应的共享卷积特征图至连接点预测模块；连接点预测模块根据共享卷积特征图输出预览图像对应的候选连接点，并传输给线段采样模块；线段采样模块根据候选连接点预测出预览图像包含的线条。
可选地，变换矩阵至少包括旋转矩阵、单应性矩阵。
可选地，构图模块，还用于获取预览图像中包含的元素的轮廓线；根据构图辅助线、以及预览图像中包含的元素的轮廓线，生成适合待拍摄场景的拍摄轮廓线。发送模块，还用于向终端设备发送拍摄轮廓线。
可选地，所述装置还包括：构图优化模块；接收模块，还用于接收来自终端设备的待拍摄场景对应的照片。构图优化模块，用于对所述照片进行构图优化，得到所述照片对应的至少一张构图优化后的照片。发送模块，还用于向终端设备发送所述至少一张构图优化后的照片。
示例性地，云端包括线条检测网络、图像矫正模块、显著性检测模块、以及美学评分网络。构图优化模块，具体用于将所述照片输入线条检测网络，通过线条检测网络检测所述照片中包含的线条；通过图像矫正模块根据所述照片中包含的线条，确定对所述照片进行矫正所需的变换矩阵，并采用变换矩阵对所述照片进行矫正，得到矫正后的照片；通过显著性检测模块对所述矫正后的照片进行显著性检测，得到所述矫 正后的照片的显著性结果；将所述矫正后的照片、以及显著性结果输入美学评分网络，得到美学评分网络输出的多张候选裁切图、以及每张所述候选裁切图对应的评分；确定多张候选裁切图中评分最高的至少一张候选裁切图为构图优化后的照片。
一些实施例中，接收模块接收到的来自终端设备的初始照片为非人像照片，如：可以是一些风景/风光照片。
第十方面，本申请实施例提供一种电子设备，包括：处理器，用于存储所述处理器可执行指令的存储器；所述处理器被配置为执行所述指令时，使得所述电子设备实现如第八方面及第八方面中的任意一种实现方式中所述的方法。
该电子设备可以是云端服务器、服务器集群、云平台等。
第十一方面，本申请实施例提供一种计算机可读存储介质，其上存储有计算机程序指令；当计算机程序指令被电子设备执行时，使得电子设备实现如第八方面及第八方面中的任意一种实现方式中所述的方法。
第十二方面，本申请实施例还提供一种计算机程序产品，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，使得电子设备实现前述第八方面及第八方面中的任意一种实现方式中所述的方法。
上述第九方面至第十二方面所具备的有益效果，可参考第八方面及第八方面中的任意一种实现方式中所述，在此不再赘述。
第十三方面，本申请实施例提供一种拍照方法，应用于端云协同系统。端云协同系统包括终端设备和云端，终端设备通过无线网络与云端连接。所述方法包括：
终端设备获取待编辑照片和待编辑照片对应的场景信息；终端设备向云端发送所述待编辑照片和场景信息。云端根据待编辑照片和场景信息，对待编辑照片进行构图优化，得到待编辑照片对应的至少一张构图优化后的照片，并将所述至少一张构图优化后的照片发送给终端设备。终端设备显示待编辑照片、以及所述至少一张构图优化后的照片。针对终端设备显示的上述照片，用户对构图满意的第一照片进行保存操作。终端设备响应于用户的操作，选择第一照片。终端设备向云端发送第一照片。云端根据第一照片获取用户的美学偏好。其中，第一照片包括初始照片、以及所述至少一张构图优化后的照片中的一张或多张。
一些实施例中，云端包括线条检测网络、图像矫正模块、显著性检测模块、以及美学评分网络；所述云端对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片，包括：云端将所述待编辑图像输入所述线条检测网络，通过所述线条检测网络检测所述待编辑图像中包含的线条；云端通过所述图像矫正模块根据所述待编辑图像中包含的线条，确定对所述预览图像进行矫正所需的变换矩阵，并采用所述变换矩阵对所述待编辑图像进行矫正，得到矫正后的待编辑图像。
一些实施例中，所述线条检测网络包括：主干网络、连接点预测模块、线段采样模块、以及线段校正模块；所述线条检测网络检测所述待编辑图像中包含的线条，包括：将所述待编辑图像输入主干网络后，所述主干网络提取所述待编辑图像中的特征，输出所述待编辑图像对应的共享卷积特征图至所述连接点预测模块；所述连接点预测模块根据所述共享卷积特征图输出所述待编辑图像对应的候选连接点，并传输给所述 线段采样模块；所述线段采样模块根据所述候选连接点预测出所述待编辑图像包含的线条。
本实施例中，终端设备将用户拍摄的照片或已有照片上传至云端，云端基于美学评分网络向终端设备返回构图优化后的照片，终端设备再将用户的拍照构图选择结果上传至云端，云端对美学评分网络进行迁移训练，获取用户的美学偏好。这种端云协同的拍照构图私人定制方法，可以依靠云端的强算力，有效降低终端设备硬件要求，持续推荐更加符合用户习惯的拍照构图，减少用户学习成本，提升用户拍照构图体验。
第十四方面，本申请实施例提供一种拍照方法，所述方法应用于终端设备，终端设备通过无线网络与云端连接。所述方法包括：终端设备响应于用户的操作，获取待编辑照片和待编辑照片对应的场景信息。终端设备向所述云端发送待编辑照片和场景信息，待编辑照片和场景信息用于所述云端对待编辑照片进行构图优化，得到待编辑照片对应的至少一张构图优化后的照片。终端设备接收所述云端发送的所述至少一张构图优化后的照片。所述终端设备显示所述待编辑照片、以及所述至少一张构图优化后的照片。所述终端设备响应于用户的选择操作，选择所述第一照片，所述第一照片包括待编辑照片、以及所述至少一张构图优化后的照片中的一张或多张。所述终端设备向所述云端发送所述保存的第一照片，第一照片用于所述云端获取用户的美学偏好。
一些实施例中，终端设备向所述云端发送所述第一照片，包括：终端设备通过预设的卷积神经网络提取所述第一照片的图像特征；所述终端设备向所述云端发送所述第一照片的图像特征。
一些实施例中，终端设备向所述云端发送所述第一照片，包括：所述终端设备向所述云端发送所述第一照片的标识信息。
第十五方面，本申请实施例提供一种终端设备，终端设备可以包括拍照装置，该装置可以用于实现上述第十四方面所述的方法。该装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元，例如，相机模块，发送模块，接收模块，显示模块，处理模块等。
其中，相机模块，用于响应于用户的操作，获取待编辑照片和所述待编辑照片对应的场景信息。发送模块，用于向所述云端发送所述待编辑照片和场景信息，所述待编辑照片和场景信息用于所述云端对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片。接收模块，用于接收所述云端发送的所述至少一张构图优化后的照片。显示模块，用于显示所述待编辑照片、以及所述至少一张构图优化后的照片。处理模块，用于响应于用户对第一照片的选择操作，选择所述第一照片，所述第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张。所述发送模块，还用于向所述云端发送所述保存的第一照片，所述第一照片用于所述云端获取用户的美学偏好。
第十六方面，本申请实施例提供一种电子设备，包括：处理器，用于存储所述处理器可执行指令的存储器；所述处理器被配置为执行所述指令时，使得所述电子设备实现如第十四方面及第十四方面中的任意一种实现方式中所述的方法。
该电子设备可以是手机、平板电脑、可穿戴设备、车载设备、AR/VR设备、笔记 本电脑、超级移动个人计算机、上网本、个人数字助理等移动终端，或者，也可以是数码相机、单反相机/微单相机、运动摄像机、云台相机、无人机等专业的拍摄设备。
第十七方面，本申请实施例还提供一种拍照方法，方法应用于云端，云端通过无线网络与终端设备连接。所述方法包括：所述云端接收来自所述终端设备的待编辑照片和场景信息。所述云端根据所述待编辑照片和场景信息，对所述待编辑照片进行构图优化，得到待编辑照片对应的至少一张构图优化后的照片，并将所述至少一张构图优化后的照片发送给所述终端设备。云端接收来自所述终端设备的第一照片，所述第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张。所述云端根据所述第一照片，获取用户的美学偏好。
第十八方面，本申请实施例提供一种电子设备，该装置可以用于实现上述第十七方面所述的方法。该装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元，例如，接收模块，处理模块，发送模块等。
其中，接收模块，用于接收来自所述终端设备的待编辑照片和场景信息。处理模块，用于根据所述待编辑照片和场景信息，对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片。发送模块，用于将所述至少一张构图优化后的照片发送给所述终端设备。所述接收模块，还用于接收来自所述终端设备的第一照片，所述第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张。所述处理模块，还用于所述云端根据所述第一照片，获取用户的美学偏好。
第十九方面，本申请实施例提供一种电子设备，包括：处理器，用于存储所述处理器可执行指令的存储器；所述处理器被配置为执行所述指令时，使得所述电子设备实现如第十七方面及第十七方面中的任意一种实现方式中所述的方法。
该电子设备可以是云端服务器、服务器集群、云平台等。
第二十方面，本申请实施例提供一种端云协同系统，包括如第十五方面及第十五方面中的任意一种实现方式中所述的终端设备和如第十七方面及第十七方面中的任意一种实现方式中所述的云端；或者，包括如第十六方面及第十六方面中的任意一种实现方式中所述的终端设备和如第十八方面及第十八方面中的任意一种实现方式中所述的云端。
第二十一方面，本申请实施例提供一种计算机可读存储介质，其上存储有计算机程序指令；当计算机程序指令被电子设备执行时，使得电子设备实现如第十四方面及第十四方面中的任意一种实现方式中所述的方法；或者，使得电子设备实现如第十七方面及第十七方面中的任意一种实现方式中所述的方法。
第二十二方面，本申请实施例还提供一种计算机程序产品，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，使得电子设备实现前述第十四方面及第十四方面中的任意一种实现方式中所述的方法；或者，得电子设备实现如第十七方面及第十七方面中的任意一种实现方式中所述的方法。
上述第十三方面至第二十二方面所具备的有益效果，可以参考前述第一方面至第 十二方面及其任意一种实现方式中所述，在此不再赘述。
第二十三方面，本申请实施例提供一种拍照方法，应用于端云协同系统。端云协同系统包括终端设备和云端，终端设备通过无线网络与云端连接。所述方法包括：
终端设备响应于用户的编辑操作，对待编辑照片进行编辑，获取编辑后的照片。终端设备向所述云端发送待编辑照片和编辑后的照片。云端根据待编辑照片和编辑后的照片，获取用户的美学偏好，并根据用户定制的拍照风格优化网络获取基础画质优化参数。云端向并将基础画质优化参数发送给终端设备。终端设备接收到基础画质优化参数后，更新本地的基础画质参数。
本实施例中，终端设备将用户编辑前后的照片上传到云端，对云端部署的拍照风格优化网络进行训练，迁移得到用户定制的拍照风格优化网络，并同步更新终端设备侧的基础画质算法。这种端云协同的拍照风格私人定制方法，可以依靠云端的强算力，有效降低终端设备硬件要求，利用云端对基础画质算法的建模及更新，持续推荐更加符合用户习惯的拍照风格，提升用户拍照体验。第二十四方面，本申请实施例提供一种拍照方法，所述方法应用于终端设备，终端设备通过无线网络与云端连接。所述方法包括：
所述终端设备响应于用户的编辑操作，获取初始照片和编辑后的第一照片。所述终端设备向所述云端发送所述初始照片和第一照片。所述云端根据所述初始照片和第一照片，获取用户定制的拍照风格优化网络，并根据用户定制的拍照风格优化网络获取基础画质优化参数。所述云端向并将所述基础画质优化参数发送给所述终端设备。所述终端设备根据所述基础画质优化参数更新本地的基础画质参数。
第二十五方面，本申请实施例提供一种终端设备，终端设备可以包括拍照装置，该装置可以用于实现上述第二十四方面所述的方法。该装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元，例如，相机模块，发送模块，接收模块，存储模块等。
其中，相机模块，用于响应于用户的操作，获取初始照片和编辑后的第一照片。发送模块，用于向云端发送初始照片和第一照片。接收模块，用于接收来自云端的基础画质优化参数。存储模块，用于根据基础画质优化参数更新本地的基础画质参数。
第二十六方面，本申请实施例提供一种电子设备，包括：处理器，用于存储所述处理器可执行指令的存储器；所述处理器被配置为执行所述指令时，使得所述电子设备实现如第二十四方面及第二十四方面中的任意一种实现方式中所述的方法。
该电子设备可以是手机、平板电脑、可穿戴设备、车载设备、AR/VR设备、笔记本电脑、超级移动个人计算机、上网本、个人数字助理等移动终端，或者，也可以是数码相机、单反相机/微单相机、运动摄像机、云台相机、无人机等专业的拍摄设备。
第二十七方面，本申请实施例还提供一种拍照方法，所述方法应用于云端，云端通过无线网络与终端设备连接；所述方法包括：
所述云端接收来自终端设备的初始照片和编辑后的第一照片。所述云端根据所述初始照片和第一照片，获取用户定制的拍照风格优化网络，并根据用户定制的拍照风格优化网络获取基础画质优化参数。所述云端将所述基础画质优化参数发送给所述终 端设备。
第二十八方面，本申请实施例提供一种电子设备，该装置可以用于实现上述第二十七方面所述的方法。该装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元，例如，接收模块，处理模块，发送模块等。
第二十九方面，本申请实施例提供一种电子设备，包括：处理器，用于存储所述处理器可执行指令的存储器；所述处理器被配置为执行所述指令时，使得所述电子设备实现如第二十七方面及第二十七方面中的任意一种实现方式中所述的方法。
该电子设备可以是云端服务器、服务器集群、云平台等。
第三十方面，本申请实施例提供一种端云协同系统，包括如第二十五方面及第二十五方面中的任意一种实现方式中所述的终端设备和如第二十八方面及第二十八方面中的任意一种实现方式中所述的云端；或者，包括如第二十六方面及第二十六方面中的任意一种实现方式中所述的终端设备和如第二十九方面及第二十九方面中的任意一种实现方式中所述的云端。
第三十一方面，本申请实施例提供一种计算机可读存储介质，其上存储有计算机程序指令；当计算机程序指令被电子设备执行时，使得电子设备实现如第二十四方面及第二十四方面中的任意一种实现方式中所述的方法；或者，使得电子设备实现如第二十七方面及第二十七方面中的任意一种实现方式中所述的方法。
第三十二方面，本申请实施例还提供一种计算机程序产品，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，使得电子设备实现前述第二十四方面及第二十四方面中的任意一种实现方式中所述的方法；或者，使得电子设备实现如第二十七方面及第二十七方面中的任意一种实现方式中所述的方法。
上述第二十三方面至第三十二方面所具备的有益效果，可以参考前述第一方面至第十二方面及其任意一种实现方式中所述，在此不再赘述。
应当理解的是，本申请中对技术特征、技术方案、有益效果或类似语言的描述并不是暗示在任意的单个实施例中可以实现所有的特点和优点。相反，可以理解的是对于特征或有益效果的描述意味着在至少一个实施例中包括特定的技术特征、技术方案或有益效果。因此，本说明书中对于技术特征、技术方案或有益效果的描述并不一定是指相同的实施例。进而，还可以任何适当的方式组合本实施例中所描述的技术特征、技术方案和有益效果。本领域技术人员将会理解，无需特定实施例的一个或多个特定的技术特征、技术方案或有益效果即可实现实施例。在其他实施例中，还可在没有体现所有实施例的特定实施例中识别出额外的技术特征和有益效果。
附图说明
图1示出了一种手机拍摄界面的示意图；
图2示出了本申请实施例提供的端云协同系统的结构示意图；
图2A示出了本申请实施例中云端对预览图像进行矫正的流程示意图；
图2B示出了本申请实施例提供的线条检测网络的结构示意图；
图2C示出了本申请实施例提供的图像矫正的效果示意图；
图2D示出了本申请实施例提供的图像矫正的另一效果示意图；
图3示出了本申请实施例提供的终端设备的结构示意图；
图4示出了本申请实施例提供的拍照方法的流程示意图；
图5示出了本申请实施例提供的一种拍摄场景的示意图；
图6示出了本申请实施例提供的拍摄界面的示意图；
图7示出了本申请实施例提供的拍摄界面的另一示意图；
图8示出了本申请实施例提供的一种拍照操作的示意图；
图9示出了本申请实施例提供的拍摄界面的又一示意图；
图10示出了本申请实施例提供的设置界面的示意图；
图11示出了本申请实施例提供的拍摄界面的又一示意图；
图12示出了本申请实施例提供的拍摄界面的又一示意图；
图13示出了本申请实施例提供的拍摄界面的又一示意图；
图14示出了本申请实施例提供的拍照方法的另一流程示意图；
图15示出了本申请实施例中云端对初始照片进行构图优化的流程示意图；
图16示出了本申请实施例提供的回归原理示意图；
图17示出了本申请实施例提供的构图优化的示意图；
图18示出了本申请实施例提供的照片显示界面的示意图；
图19示出了本申请实施例提供的拍照装置的结构示意图；
图20示出了本申请实施例提供的电子设备的另一结构示意图；
图21示出了本申请实施例提供的电子设备的又一结构示意图；
图22示出了本申请实施例提供的拍照方法的另一流程示意图；
图23示出了本申请实施例提供的拍照方法的另一流程示意图；
图24示出了本申请实施例提供的照片编辑界面的示意图；
图25示出了本申请实施例提供的拍照方法的另一流程示意图；
图26A示出了本申请实施例提供的照片编辑界面的另一示意图之一；
图26B示出了本申请实施例提供的照片编辑界面的另一示意图之二；
图26C示出了本申请实施例提供的照片编辑界面的另一示意图之三；
图27A示出了本申请实施例提供的照片编辑界面的又一示意图之一；
图27B示出了本申请实施例提供的照片编辑界面的又一示意图之二；
图27C示出了本申请实施例提供的照片编辑界面的又一示意图之三；
图28示出了本申请实施例提供的拍照装置的另一结构示意图；
图29示出了本申请实施例提供的电子设备的另一结构示意图；
图30示出了本申请实施例提供的拍照装置的另一结构示意图；
图31示出了本申请实施例提供的电子设备的另一结构示意图。
具体实施方式
以下实施例中所使用的术语只是为了描述特定实施例的目的，而并非旨在作为对 本申请的限制。如在本申请的说明书和所附权利要求书中所使用的那样，单数表达形式“一个”、“一种”、“所述”、“上述”、“该”和“这一”旨在也包括例如“一个或多个”这种表达形式，除非其上下文中明确地有相反指示。还应当理解，在本申请以下各实施例中，“至少一个”、“一个或多个”是指一个或两个以上(包含两个)。字符“/”一般表示前后关联对象是一种“或”的关系。“和/或”用于描述关联对象的关联关系，表示可以存在三种关系。例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。
在本说明书中描述的参考“一个实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此，在本说明书中的不同之处出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是必然都参考相同的实施例，而是意味着“一个或多个但不是所有的实施例”，除非是以其他方式另外特别强调。术语“包括”、“包含”、“具有”及它们的变形都意味着“包括但不限于”，除非是以其他方式另外特别强调。术语“连接”包括直接连接和间接连接，除非另外说明。
以下，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。
在本申请实施例中，“示例性地”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性地”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性地”或者“例如”等词旨在以具体方式呈现相关概念。
随着手机拍照技术的飞速发展，使用手机拍照已经成为了人类生活中不可或缺的一部分。当用户启动手机的相机进行拍照时，手机可以采集待拍摄场景对应的预览图像，并将预览图像显示在相机的拍摄界面中，以使得用户可以看到待拍摄场景的预览画面(即预览图像在拍摄界面中显示出的画面)。用户可以根据拍摄界面中显示的预览画面对感光度、光圈等拍摄参数、以及构图方式进行调整。对构图方式进行调整可以是指对预览画面中的各元素(如：人物、建筑、动物等)在预览画面中的位置和比例进行调整。在完成感光度、光圈等拍摄参数、以及构图方式的调整后，用户可以点击相机的拍照按钮，手机可以响应于用户对拍照按钮的点击操作，按照预览画面的感光度、光圈等拍摄参数，以及预览画面的构图方式，对待拍摄场景进行拍照，得到待拍摄场景对应的照片。
其中，用户在使用手机进行拍照时，对构图方式的调整会影响到最终拍摄到的照片质量。例如，好的构图可以让拍出的照片更加出色，而错误的构图会导致拍摄出的照片达不到预期效果。更甚至，在拍照时进行较好的构图往往可以化腐朽为神奇，弥补用户拍照经验不足的缺点。
目前，为了能够在用户使用手机进行拍照时辅助用户完成较好的构图，手机在相机启动运行时，可以在相机的拍摄界面中显示预设的参考线，供用户根据参考线完成构图。例如，图1示出了一种手机拍摄界面的示意图。如图1所示，手机在相机启动 运行时，相机的拍摄界面中显示的参考线为九宫格网格线101。用户使用手机进行拍照时，可以参考拍摄界面中显示的九宫格网格线101对手机的拍摄位置、角度等进行调整，从而完成三分法构图、对角线构图、对称构图等多种不同方式的构图。
以三分法构图为例，可以看到图1中所示的九宫格网格线101以“井”字型排列，将预览画面分为九等份，其中，横竖两条线的四个交叉点一般可以认为是人的视觉兴趣点。用户在构图时可以对手机的拍摄位置、角度等进行调整，使得预览画面中的拍摄主体处于交叉点附近(或者，也可以使拍摄主体处于两条竖线附近)，从而完成三分法构图。
但是，上述通过在拍摄界面中显示预设的参考线辅助用户完成构图的方式中，参考线的引导作用有限，最终构图好坏与用户的拍摄经验和手法强相关，一些没有丰富拍摄经验的用户可能并不懂得如何根据参考线对手机的拍摄位置、角度等进行调整以完成不同方式的构图。
例如，当拍摄场景中包括天空和大海，且天空和大海具有明显的分界线时，可能更适合对称构图，具有一定拍摄经验的用户能够对手机的拍摄位置、角度等进行调整，使得天空和大海的分界线处于九宫格网格线中的两条横线的中间位置以实现对称构图。但对于没有丰富拍摄经验的用户而言，并不了解如何结合拍摄场景完成构图。
也即，目前手机通过在拍摄界面中显示预设的参考线辅助用户完成构图的方式，最终的构图效果是因人而异的，并不能帮助更多用户完成较好的构图。
在此背景技术下，本申请实施例提供了一种拍照方法，能够在用户使用具有拍照功能的终端设备进行拍照时，结合当前的拍摄场景在终端设备提供的拍摄界面中显示构图辅助线，对用户进行构图引导。其中，构图辅助线可以为用户指示适合待拍摄场景的构图方式，引导用户按照构图辅助线指示的构图方式对手机的拍摄位置、角度等进行调整以完成构图，从而可以使得没有丰富拍摄经验的用户在进行拍照时也能够完成较好的构图，用户的体验可以更好。
该方法可以应用于由终端设备和云端组成的端云协同系统。其中，端云协同的“端”指终端设备，“云”指云端，云端也可称为云服务器、远程服务器或云平台。
示例性地，图2示出了本申请实施例提供的端云协同系统的结构示意图，如图2所示，该端云协同系统可以包括：终端设备210和云端220，终端设备210可以通过无线网络与云端220连接。终端设备210具有拍照功能。
可选地，终端设备210可以是手机、平板电脑、可穿戴设备、车载设备、增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)设备、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、个人数字助理(personal digital assistant，PDA)等移动终端，或者，也可以是数码相机、单反相机/微单相机、运动摄像机、云台相机、无人机等专业的拍摄设备，本申请实施例对终端设备210的具体类型不作限制。
应当理解，当终端设备210为云台相机、无人机等拍摄设备时，终端设备210还会包括一个可以提供拍摄界面的显示设备，用于显示构图辅助线。例如，云台相机的显示设备可以是手机，航拍无人机的显示设备可以是遥控设备等。
可选地，云端220可以是计算机、服务器、或者多个服务器组成的服务器集群等，本申请对云端220的实现架构不作限制。另外，终端设备210的具体形态可以参考前述实施例中所述，不再赘述。
基于图2所示的端云协同系统，本申请实施例提供的拍照方法可以如下：
终端设备210在进行拍照时(如启动拍照应用程序后)，按照第一帧率获取待拍摄场景对应的预览图像，并在拍摄界面进行显示。同时，终端设备210按照第一频率，通过传感器采集待拍摄场景的场景信息。在前述过程中，终端设备210按照第二频率向云端220发送获取到的预览图像和场景信息。云端220每一次接收到预览图像和场景信息后，根据本次接收到的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线，并将构图辅助线发送给终端设备210。终端设备210接收到构图辅助线后，将构图辅助线与正在显示的预览图像一同显示在拍摄界面。用户可以参考拍摄界面中显示的构图辅助线，对终端设备210的拍摄位置、角度等进行调整以完成构图。完成构图后，用户可以对终端设备210执行拍照操作，以触发终端设备210进行拍照。
其中，第二频率的大小小于或等于第一帧率和第一频率中的最小值。第一帧率和第一频率的大小可以相同或不同，本申请对第一帧率和第一频率的大小均不作限制。
示例性地，在一种可能的设计中，第一帧率可以是30帧每秒(frames per second，FPS)。第一帧率和第一频率的大小相同时，第一频率可以为30次/秒。第一帧率和第一频率的大小不同时，第一频率可以为10次/秒、15次/秒、35次/秒等，第一频率可以小于第一帧率，也可以大于第一帧率。第二频率可以参考第一频率，不再举例说明。
一些实施例中，终端设备210上可以配置有位置传感器、气压传感器，温度传感器，环境光传感器等传感器。与终端设备210配置的传感器相对应，终端设备210通过传感器采集的待拍摄场景的场景信息可以包括：待拍摄场景对应的位置信息、气压信息、温度信息、光强信息等。本申请实施例对终端设备210中拍摄的传感器的数量和类型、以及终端设备210通过传感器采集的待拍摄场景的场景信息均不作限制。
针对每一次接收到的预览图像和场景信息，云端220根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线的过程如下：
一些实施例中，云端220预设有根据预览图像中包含的元素、以及不同元素的位置和所占比例，生成适合待拍摄场景的构图方式对应的构图辅助线的匹配规则。该匹配规则可以是根据摄影规则人为定义的。例如，可以人为定义哪些拍摄场景适合哪种构图方式，并为该拍摄场景设置该构图方式对应的构图辅助线，然后将前述匹配规则配置到云端中。针对每一次接收到的预览图像和场景信息，云端220首先可以根据预览图像和场景信息，识别出预览图像中包含的元素，并记录预览图像中不同元素的位置和所占比例。然后，云端220可以根据预览图像中包含的元素，以及不同元素的位置和所占比例，在前述预设的匹配规则中，确定出适合待拍摄场景的构图方式对应的构图辅助线。
示例性地，预览图像中包含的元素可以是天空、海水、草地、人物等。元素的位置是指预览图像中元素所在区域的像素点坐标，元素所占的比例是指预览图像中元素 所在区域的像素点数量占整个预览图像的像素点数量的比值。
可选地，本申请实施例中，云端220根据预览图像和场景信息，识别预览图像中包含的元素，并记录预览图像中不同元素的位置和所占比例的步骤，可以包括：云端220采用第一方法对预览图像进行分割，然后基于分割结果、结合场景信息识别预览图像中包含的元素，并记录预览图像中不同元素的位置和所占比例的步骤。其中，场景信息可以用于辅助云端220基于分割结果快速识别预览图像中包含的元素。例如，当场景信息包括的位置信息为海边时，可以表示预览图像中可能含有海水，能够辅助云端220基于分割结果快速识别出预览图像中是否包含海水。类似地，气压信息、光线信息等均可以用于辅助云端220基于分割结果快速识别预览图像中包含的元素，不再一一举例。
示例性地，上述第一方法可以是基于边缘检测的方法、基于小波变换的方法、基于深度学习的方法等。本申请实施例中，考虑到性能要求高，精度要求较低，第一方法可以尽量选用传统分割方法或者模型较小的方法，例如，云端220可以采用深度学习分割网络U-NET对预览图像进行分割。但需要说明，本申请对第一方法的具体类型并不作限制。
示例性地，云端220预设的根据预览图像中包含的元素、以及不同元素的位置和所占比例，生成适合待拍摄场景的构图方式对应的构图辅助线的匹配规则，可以包含下述a)至e)中的一种或多种。云端220可以结合下述a)至e)中的一种或多种，根据预览图像中包含的元素，以及不同元素的位置和所占比例，确定出适合待拍摄场景的构图方式对应的构图辅助线。
a)如果识别得到预览图像中包含天空、海水、草地、山脉等元素的其中两种，且两种元素之间有明显的分界线，比如海平面、天际线等，则确定适合待拍摄场景的构图方式为对称构图，并结合待拍摄场景生成对称构图方式对应的构图辅助线。
b)如果识别得到预览图像中包含天空、海水、草地、山脉等元素的其中三种，且相邻的两种元素之间有明显的分界线，比如海平面、天际线等，则确定适合待拍摄场景的构图方式为三分构图，并结合待拍摄场景生成三分构图方式对应的构图辅助线。
c)如果识别得到预览图像中包含明显的主体，比如需要特写的人物、动物、树木、船只等，则云端可以确定适合待拍摄场景的构图方式为黄金分割构图或中心构图，并结合待拍摄场景(如：以主体为目标)生成黄金分割构图方式或中心构图方式对应的构图辅助线。
d)如果识别得到预览图像中包含明显的弯曲道路、桥梁、铁轨等，则确定适合待拍摄场景的构图方式为引导线构图，并结合待拍摄场景(如：沿着弯曲道路、桥梁、铁轨等)生成引导线构图方式对应的构图辅助线。
e)如果识别得到预览图像中包含天空和建筑，则根据预览图像中天空和建筑分别所占的比例确定适合待拍摄场景的构图方式为二八分构图、三七分构图、五五分构图等，并结合待拍摄场景生成前述二八分构图、三七分构图、五五分构图等构图方式对应的构图辅助线。
可以看到，上述匹配规则中，根据预览图像中包含的元素、以及不同元素的位置 和所占比例，可以将预览图像对应的待拍摄场景划分为不同的类型，针对每种类型的待拍摄场景，可以人为定义适合该待拍摄场景的构图方式对应的构图辅助线。
应当理解，上述a)至e)所述的匹配规则均仅为示例性说明，本申请实施例对拍摄场景和构图方式之间的匹配规则并不作限制。例如，云端220还可以包括更多适用于不同拍摄场景的构图方式。
另外，还需要说明的是，云端220根据预览图像生成的适合待拍摄场景的构图方式对应的构图辅助线，其本质可以是预览图像中的多个像素点坐标组成的坐标数据集合，坐标数据集合中的多个像素点坐标对应的像素点连接起来即为构图辅助线。
可选地，本申请实施例中，在进行人为定义匹配规则时，对任意一种类型的待拍摄场景，若认为适合该种待拍摄场景的构图方式只有一种，则可以在匹配规则中为这种类型的待拍摄场景设置这一种符合的构图方式。若认为适合该种待拍摄场景的构图方式可以包括多种，则可以在匹配规则中将多种构图方式均设置为适合这种类型的待拍摄场景的构图方式。也即，上述人为定义的匹配规则中，每种类型的待拍摄场景可以对应一种构图方式，也可以对应多种构图方式，本申请也不作限制。
对于任意一种类型的待拍摄场景而言，当匹配规则中该种类型的待拍摄场景对应的构图方式包括一种时，云端220在获取到该种类型的待拍摄场景对应的预览图像后，会直接根据匹配规则生成这一种构图方式对应的构图辅助线。
当匹配规则中该种类型的待拍摄场景对应的构图方式包括多种时，一些实施方式中，云端220在获取到该种类型的待拍摄场景对应的预览图像后，会根据匹配规则从符合该待拍摄场景的多种构图方式中随机选择一种构图方式，并生成这种构图方式对应的构图辅助线，发送给终端设备210。
另外一些实施方式中，在进行人为定义匹配规则时，若认为适合某种待拍摄场景的构图方式包括多种，则在匹配规则中将多种构图方式均设置为适合这种类型的待拍摄场景的构图方式的同时，还可以人为按照摄影经验、或者已有的摄影案例等为多种构图方式分别一一设置对应的概率，不同的构图方式对应的概率不同。云端220从符合待拍摄场景的多种构图方式中选择一种构图方式时，会根据多种构图方式分别对应的概率，选择对应的概率最大的一种构图方式，并生成这种构图方式对应的构图辅助线，发送给终端设备210。
例如，对于某种类型的待拍摄场景而言，若适合构图方式1、构图方式2、构图方式3三种构图方式，但已有的一些同样类型的拍摄场景的图片中，构图方式1的占比为二分之一，构图方式2的占比为八分之三，构图方式3的占比为八分之一，则人为定义匹配规则时，可以将这三种构图方式均设置为适合该种类型的待拍摄场景的构图方式，但同时会根据这三种构图方式的占比分别设置构图方式1的概率为二分之一、构图方式2的概率为八分之三，构图方式3的概率为八分之一。云端从符合该种类型的待拍摄场景的构图方式1、构图方式2、构图方式3中选择一种构图方式时，会选择概率为二分之一的构图方式1。
另外一些实施例中，云端220预设有训练好的人工智能(artificial intelligence，AI)网络。该AI网络具有根据预览图像的显著性结果和场景信息，生成适合待拍摄场景的 构图方式的功能。针对每一次接收到的预览图像和场景信息，云端220首先可以对预览图像进行显著性检测，提取预览图像的显著性结果；然后可以将预览图像的显著性结果和场景信息输入该AI网络。该AI网络可以输出预览图像对应的多种构图方式的概率分布(概率分布可以参考前述实施例中所述的不同构图方式对应的概率)。通过该概率分布可以得知，哪种构图方式更适合预览图像对应的待拍摄场景。云端220可以根据AI网络输出的预览图像对应的多种构图方式的概率分布，从多种构图方式中选择概率最大的一种构图方式，并生成该种构图方式对应的构图辅助线。
可以理解的，对某种类型的待拍摄场景而言，AI网络输出的概率分布中，不适合该类型的待拍摄场景的构图方式的概率为0或接近于0。
示例性地，上述AI网络可以通过采集对多种拍摄场景进行拍摄得到的大量样本图像、以及每张样本图像对应的拍摄场景的场景信息，然后使用样本图像和场景信息对神经网络进行训练得到。例如，首先可以获取对多种拍摄场景进行拍摄得到的大量样本图像、以及每张样本图像对应的拍摄场景的场景信息(场景信息具体参见前述实施例)，并对每张样本图像进行显著性检测，生成每张样本图像的显著性结果。同时，人为标记每张样本图像分别对应的构图方式。然后，将每张样本图像的显著性结果、以及对应的拍摄场景的场景信息作为输入，样本图像上标记的构图方式作为输出对神经网络进行训练，得到AI网络。训练好的AI网络可以学习到样本图像对应的拍摄场景的场景信息、样本图像中包含的元素、以及不同元素的位置和所占比例，与样本图像的构图方式的映射关系。从而，AI网络具有根据预览图像的显著性结果和场景信息，生成适合待拍摄场景的构图方式对应的功能。
上述基于匹配规则或AI网络的两种实现方式中，当符合待拍摄场景的构图方式包括多种时，云端220均会从多种构图方式中选择一种构图方式，并生成这种构图方式对应的构图辅助线，发送给终端设备210。
可选地，还有一些实施例中，当符合待拍摄场景的构图方式包括多种时，云端220也可以从符合待拍摄场景的多种构图方式中选择至少两种构图方式(如选择概率最大的两种构图方式)，并生成前述至少两种构图方式对应的构图辅助线，发送给终端设备210。终端设备210在接收到前述至少两种构图方式对应的构图辅助线后，可以在拍摄界面主要显示其中一种构图方式对应的构图辅助线，同时，将其他构图方式对应的构图辅助线，以分屏的方式、或者小窗口显示的方式、又或者标识控件的方式显示在拍摄界面中，以使得用户可以主动进行切换操作，触发终端设备210将拍摄界面主要显示的构图辅助线，在前述至少两种构图方式对应的构图辅助线中进行切换，如可以从构图辅助线1切换至构图辅助线2。本申请对终端设备210显示多种构图辅助线的具体方式不作限制。
应当理解，针对终端设备210每一次向云端220发送的预览图像和场景信息，终端设备210都可以对应接收到一次构图辅助线。终端设备210从第二次接收到构图辅助线开始，对于每一次接收到的构图辅助线，终端设备210将本次接收到的构图辅助线在拍摄界面显示，实际上可以理解为对拍摄界面中显示的上一次接收到的构图辅助线进行了更新。
假设终端设备210将某一次接收到的构图辅助线与正在显示的预览图像一同显示在拍摄界面时，当前正在显示的预览图像为终端设备210在t时刻所获取的，则该构图辅助线是云端220根据终端设备210在(t-dt)时刻所发送的预览图像和场景信息所生成的。其中，t大于0，dt大于0，dt表示从终端设备210向云端220发送预览图像和场景信息，至云端220向终端210返回构图辅助线的整个处理过程的时延。
本申请实施例提供的拍照方法中，终端设备210显示的构图辅助线可以为用户指示适合待拍摄场景的构图方式，引导用户按照构图辅助线指示的构图方式对终端设备210的拍摄位置、角度等进行调整以完成构图，从而可以使得没有丰富拍摄经验的用户在进行拍照时也能够完成较好的构图，用户的体验可以更好。通过该拍照方法可以使用户更轻松地知道通过哪种构图方式进行构图，不需要自己判断构图方式，也不要复杂的构图操作。
可选地，本申请实施例中，云端220每一次接收到预览图像和场景信息后，在根据本次接收到的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线之前，还可以先对预览图像进行矫正。云端220可以根据矫正后的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线。
示例性地，图2A示出了本申请实施例中云端对预览图像进行矫正的流程示意图。如图2A所示，云端220可以包括：线条检测网络和图像矫正模块。
云端220对预览图像的矫正流程可以包括：云端220将预览图像输入线条检测网络，通过线条检测网络检测预览图像中包含的水平线、竖直线、建筑物的轮廓线等线条。线条检测网络检测得到预览图像中包含的线条后，图像矫正模块根据预览图像中包含的线条确定对预览图像进行矫正所需的变换矩阵，并采用该变换矩阵对预览图像进行矫正，得到矫正后的预览图像。
可选地，线条检测网络可以为端到端训练的AI网络模型。例如，图2B示出了本申请实施例提供的线条检测网络的结构示意图。如图2B所示，线条检测网络可以包括：主干网络(backbone)、连接点预测模块(junction proposal module)、线段采样模块(line sampling module)、以及线段校正模块(line verification module)。
将预览图像输入主干网络后，主干网络可以提取预览图像中的特征，输出预览图像对应的共享卷积特征图至连接点预测模块。连接点预测模块可以根据共享卷积特征图输出预览图像对应的候选连接点，并传输给线段采样模块。线段采样模块可以根据候选连接点预测出预览图像包含的线条(或称为线段)。线段校正模块可以对预测出的线条进行分类，最终输出检测到的预览图像中包含的线条。
可选地，图像矫正模块根据预览图像中包含的线条确定出的对预览图像进行矫正所需的变换矩阵，可以包括但不限于旋转(rotation)矩阵、单应性矩阵(homography matrix)。
例如，当检测到预览图像中包含的水平线和/或竖直线倾斜时，图像矫正模块可以确定变换矩阵为旋转(rotation)矩阵，并使用旋转(rotation)矩阵对预览图像进行变换调整。如，图2C示出了本申请实施例提供的图像矫正的效果示意图。对于图2C中的(a)所示的预览图像，线条检测网络可以检测到其包含的水平线(如：天空和海水 的交界线)向右倾斜，图像矫正模块可以确定对图2C中的(a)所示的预览图像进行矫正所需的旋转矩阵，并使用该旋转矩阵对预览图像进行变换调整。对图2C中的(a)所示的预览图像进行矫正后得到的矫正后的预览图像可以如图2C中的(b)所示。
又例如，当检测到预览图像中包含的建筑存在透视问题时，图像矫正模块可以确定变换矩阵为单应性矩阵(homography matrix)，并使用单应矩阵对预览图像进行变换调整。如，图2D示出了本申请实施例提供的图像矫正的另一效果示意图。对于图2D中的(a)所示的预览图像，线条检测网络可以检测到其包含的建筑存在透视问题，图像矫正模块可以确定对图2D中的(a)所示的预览图像进行矫正所需的单应性矩阵，并使用该单应性矩阵对预览图像进行变换调整。对图2D中的(a)所示的预览图像进行矫正后得到的矫正后的预览图像可以如图2D中的(b)所示。
本申请实施例中，云端220对预览图像进行矫正，可以使得云端220根据矫正后的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线时，能够更加准确地识别到预览图像中包含的元素、以及不同元素的位置和所占比例，进而使生成的辅助线更加符合待拍摄场景对应的构图方式。
需要说明的是，上述图2中示例性给出了一个终端设备210。但应当理解，该端云协同系统中的终端设备210可以包括一个或多个，多个终端设备210可以相同，也可以不相同或部分相同，在此均不作限制。本申请实施例提供的拍照方法是针对每个终端设备210与云端220之间进行交互实现拍照的过程。
另外，本申请实施例中所述的构图辅助线也可以被称为参考线、构图参考线、构图线等，在此对构图辅助线的名称并不作限制。
下面以上述图2所示的端云协同系统中的终端设备210为手机为例，结合用户使用手机进行拍照的场景，对本申请实施例提供的拍照方法进行示例性说明。
需要说明的是，本申请实施例虽然是以终端设备210为手机为例进行说明，但应当理解，本申请实施例提供的拍照方法同样适用于上述其他具有拍照功能的终端设备，本申请对该终端设备210的具体类型并不作限制。
示例性地，以终端设备为手机为例，图3示出了本申请实施例提供的终端设备的结构示意图。如图3所示，手机可以包括处理器310，外部存储器接口320，内部存储器321，通用串行总线(universal serial bus，USB)接口330，充电管理模块340，电源管理模块341，电池342，天线1，天线2，移动通信模块350，无线通信模块360，音频模块370，扬声器370A，受话器370B，麦克风370C，耳机接口370D，传感器模块380，按键390，马达391，指示器392，摄像头393，显示屏394，以及用户标识模块(subscriber identification module，SIM)卡接口395等。
处理器310可以包括一个或多个处理单元，例如：处理器310可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
其中，控制器可以是手机的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成读取指令和执行指令的控制。
处理器310中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器310中的存储器为高速缓冲存储器。该存储器可以保存处理器310刚用过或循环使用的指令或数据。如果处理器310需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器310的等待时间，因而提高了系统的效率。
在一些实施例中，处理器310可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，SIM接口，和/或USB接口等。
外部存储器接口320可以用于连接外部存储卡，例如Micro SD卡，实现扩展手机的存储能力。外部存储卡通过外部存储器接口320与处理器310通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器321可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器310通过运行存储在内部存储器321的指令，从而执行手机的各种功能应用以及数据处理。内部存储器321可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储手机使用过程中所创建的数据(比如图像数据，电话本等)等。此外，内部存储器321可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
充电管理模块340用于从充电器接收充电输入。充电管理模块340为电池342充电的同时，还可以通过电源管理模块341为手机供电。电源管理模块341用于连接电池342，充电管理模块340，以及处理器310。电源管理模块341也可接收电池342的输入为手机供电。
手机的无线通信功能可以通过天线1，天线2，移动通信模块350，无线通信模块360，调制解调处理器以及基带处理器等实现。天线1和天线2用于发射和接收电磁波信号。手机中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。手机在进行拍照时，可以通过无线通信功能，向云端220发送获取到的预览图像和场景信息，并接收云端220根据预览图像和场景信息发送的构图辅助线。
手机可以通过音频模块370，扬声器370A，受话器370B，麦克风370C，耳机接口370D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
传感器模块380可以包括压力传感器380A，陀螺仪传感器380B，气压传感器380C，磁传感器380D，加速度传感器380E，距离传感器380F，接近光传感器380G，指纹传 感器380H，温度传感器380J，触摸传感器380K，环境光传感器380L，骨传导传感器380M等。图3中仅示例性给出了传感器模块380包括的部分传感器，例如，传感器模块380还包括位置传感器(如GPS)。手机在拍照时，可以通过传感器模块380中的传感器，采集待拍摄场景的场景信息，如：气压信息、温度信息、位置信息(如GPS坐标)、光强信息等。
摄像头393可以包括多种类型。例如，摄像头393可以包括具有不同焦段的长焦摄像头，广角摄像头或超广角摄像头等。其中，长焦摄像头的视场角小，适用于拍摄远处小范围内的景物；广角摄像头的视场角较大；超广角摄像头的视场角大于广角摄像头，可以用于拍摄全景等大范围的画面。在一些实施例中，视场角较小的长焦摄像头可转动，从而可以拍摄不同范围内的景物。
手机可以通过摄像头393捕获待拍摄场景的原始图像(也称为RAW图或数字底片)。例如，摄像头393至少包括镜头(lens)和传感器(sensor)。在拍摄照片或者拍摄视频时，打开快门，光线可以通过摄像头393的镜头被传递到sensor上。sensor可以将通过镜头的光信号转换为电信号，再对电信号进行模数(analogue-to-digital，A/D)转换，输出对应的数字信号。该数字信号即为RAW图。之后，手机可以通过处理器(如：ISP、DSP等)对RAW图进行后续的ISP处理、以及YUV域处理等，将RAW图转化为可用于显示的图像，如：JPEG图像或高效率图像文件格式(high efficiency image file format，HEIF)图像。JPEG图像或HEIF图像可以被传输给手机的显示屏进行显示，和/或，传输给手机的存储器进行存储。从而，手机可以实现拍摄的功能。
在一种可能的设计中，sensor的感光元件可以是电荷耦合器件(charge coupled device，CCD)，sensor还包括A/D转换器。在另外一种可能的设计中，sensor的感光元件可以是互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)。
示例性地，ISP处理可以包括：坏点矫正(bad pixel correction，DPC)、RAW域降噪、黑电平矫正(black level correction，BLC)、镜头亮度矫正(lens shading correction，LSC)、自动白平衡(auto white balance，AWB)、去马赛克(demosica)颜色插值、色彩校正(color correction matrix，CCM)、动态范围压缩(dynamic range compression，DRC)、伽玛(gamma)、3D查找表(look up table，LUT)、YUV域降噪、锐化(sharpen)、增强细节(detail enhance)等。YUV域处理可以包括：高动态范围图像(high-dynamic range，HDR)的多帧配准、融合、降噪，以及提升清晰度的超分辨率(super resolution，SR)算法、美肤算法、畸变校正算法、虚化算法等。
显示屏394用于显示图像，视频等。显示屏394包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，手机可以包括1个或N个显示屏394，N为大于1的正整数。例如，显示屏394可以用于显示拍摄界面，照片播放界面等。本申请实施例中，拍摄界面可以包括预览图像、云端220发送给手机的构图辅助线。
手机通过GPU，显示屏394，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏394和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器310可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
可以理解的是，图3所示的结构并不构成对手机的具体限定。在一些实施例中，手机也可以包括比图3所示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置等。又或者，图3所示的一些部件可以以硬件，软件或软件和硬件的组合实现。
另外，当终端设备210是其他平板电脑、可穿戴设备、车载设备、AR/VR设备、笔记本电脑、UMPC、上网本、PDA等移动终端，或者，数码相机、单反相机/微单相机、运动摄像机、云台相机、无人机等专业的拍摄设备时，这些其他终端设备的具体结构也可以参考图3所示。示例性地，其他终端设备可以是在图3给出的结构的基础上增加或减少了组件，在此不再一一赘述。
还应当理解的是，终端设备210(如手机)中可以运行有一个或多个拍照应用程序，以便通过运行拍照应用程序，实现拍摄的功能。例如，该拍照应用程序可以包括系统级应用：相机。又如，该拍照应用程序还可以包括其他安装在终端设备中的能够用于拍摄的应用程序。
示例性地，图4示出了本申请实施例提供的拍照方法的流程示意图。如图4所示，本申请实施例提供的拍照方法可以包括：S401-S407。
S401、手机在启动运行拍照应用程序后，按照第一帧率获取待拍摄场景对应的预览图像，并在拍摄界面进行显示；同时，按照第一频率，通过传感器采集待拍摄场景的场景信息。
示例性地，当用户需要使用手机进行拍照时，可以先启动手机的拍照应用程序。例如，用户可以点击或触摸手机上的相机的图标，手机可以响应于用户对相机的图标的点击或触摸操作，启动运行相机(或者，用户还可以通过语音助手启动相机，不作限制)。
在启动运行拍照应用程序后，手机可以通过相机模块(如前述摄像头)采集待拍摄场景对应的RAW图。然后，手机的处理器可以对RAW图进行简单的ISP处理，得到待拍摄场景对应的YUV图。该YUV图即为待拍摄场景对应的预览图像。
或者，另外一些实施例中，在得到YUV图后，手机中的处理器还可以将YUV图转换为RGB格式的RGB图，此时，该RGB图可以作为待拍摄场景对应的预览图像。
手机在启动运行拍照应用程序后，会显示拍照应用程序提供的拍摄界面，该拍摄界面能够用于显示拍照时的预览图像，以使得用户可以看到待拍摄场景的预览画面。预览图像的刷新帧率为第一帧率，具体请参见前述实施例。
手机在获取并显示预览图像的同时，可以按照第一频率，通过传感器采集待拍摄场景的场景信息。第一频率、以及通过传感器采集待拍摄场景的场景信息的具体过程，也请参见前述实施例中所述，不再赘述。
S402、手机按照第二频率向云端发送预览图像和场景信息。
本申请实施例中，手机是在执行S401的过程中，同步执行S402的。
相应地，云端接收手机发送(或称为上传)的预览图像和场景信息。
可选地，手机向云端发送的预览图像可以为上述S401的示例性说明中提到的YUV图或者RGB图，在此不作限制。
针对每一次接收到的预览图像和场景信息，云端可以执行S403-S405所述的步骤。
S403、云端对预览图像进行矫正，得到矫正后的预览图像。
云端对预览图像进行矫正的具体实现已在前述实施例中详细说明，不再赘述。
S404、云端根据矫正后的预览图像，生成适合待拍摄场景的构图方式对应的构图辅助线。
云端根据矫正后的预览图像，生成适合待拍摄场景的构图方式对应的构图辅助线的过程，也请参见前述实施例中所述。
S405、云端向手机返回构图辅助线。
相应地，手机接收云端返回的构图辅助线。
一些实施例中，云端向手机返回的构图辅助线可以如前述实施例中所述，为预览图像中的多个像素点坐标组成的坐标数据集合。
另外一些实施例中，云端也可以向手机返回一张包含构图辅助线的图像，该图像中，构图辅助线所在区域的像素点的像素值可以为0，除构图辅助线之外的区域的像素点的像素值可以为P，P为大于0、小于或等于255的整数。从而，云端可以实现向手机返回构图辅助线。可选地，除构图辅助线之外的区域的像素点的像素值可以均为255。需要说明的是，本申请对构图辅助线所在区域以及除构图辅助线之外的区域的像素点的像素值均不作限制，如：构图辅助线所在区域的像素点的像素值也可以是0至255中的其他值，构图辅助线所在区域的像素点的像素值与除构图辅助线之外的区域的像素点的像素值不同即可。
S406、手机将构图辅助线与正在显示的预览图像显示在拍摄界面。
用户使用手机进行拍照前，可以参考拍摄界面中显示的构图辅助线，对手机的拍摄位置、角度等进行调整，从而完成适合待拍摄场景的构图。
举例说明，图5示出了本申请实施例提供的一种拍摄场景的示意图，图6示出了本申请实施例提供的拍摄界面的示意图。如图5所示，假设待拍摄场景包含天空和大海，且天空和大海之间具有明确的分界线。则手机在启动运行拍照应用程序后，可以获取图5所示的拍摄场景对应的预览图像发送给云端，该预览图像可以是YUV图或RGB图。根据前述S404中所述可知，云端能够根据该预览图像(矫正后的预览图像)确定出适合图5所示的待拍摄场景的构图方式为对称构图，并结合图5所示的待拍摄场景生成对称构图方式对应的构图辅助线。
之后，云端可以将该构图辅助线返回给手机。当手机接收到云端返回的构图辅助线后，可以将构图辅助线与正在显示的预览图像一同显示在拍摄界面。请参考图6所示，拍摄界面至少包括：正在显示的预览图像呈现出的预览画面、构图辅助线601。其中，预览画面即为待拍摄场景对应的预览画面，包含了天空和大海，且天空和大海之间具有明确的分界线。构图辅助线601即为云端结合图5所示的待拍摄场景生成的对称构图方式对应的构图辅助线，能够向用户指示适合待拍摄场景的构图方式为对称构 图。用户使用手机进行拍照前，可以参考图6所示的拍摄界面中显示的预览画面对拍摄画面进行预览，以及参考图6所示的拍摄界面中显示的构图辅助线601对手机的拍摄位置、角度等进行调整，从而在预览时完成适合待拍摄场景的对称构图。
例如，用户从图6所示的拍摄界面可以看到，预览画面中天空和大海的分界线偏高。所以，用户可以将手机向上移动和/或向后旋转一定角度，使得天空和大海的分界线与构图辅助线601接近或重合，从而完成对称构图。示例性地，图7示出了本申请实施例提供的拍摄界面的另一示意图，完成对称构图后的预览画面可以如图7中所示。
可选地，一些实施例中，手机将构图辅助线与正在显示的预览图像一同显示在拍摄界面的同时，还可以在拍摄界面中显示文字提示，提示用户按照构图辅助线引导的方式完成构图。例如，对于上述图5所示的待拍摄场景，图6所示的拍摄界面中还可以显示文字提示：“请将天空和大海的分界线移动至与构图辅助线重合”。示例性地，该文字提示也可以是由云端在生成构图辅助线时，一并生成的。云端可以将构图辅助线和文字提示一并发送给手机。需要说明的是，本申请实施例对文字提示在拍摄界面中的位置、以及文字提示的具体内容不作限制。例如，待拍摄场景不同或者构图辅助线不同时，文字提示可能不同。
用户在完成构图后，可以进行拍照操作，触发手机拍照，如：该拍照方法还包括S407。
S407、手机响应于用户的拍照操作，进行拍照。
示例性地，图8示出了本申请实施例提供的一种拍照操作的示意图。如图8所示，本申请实施例中，手机通过拍照应用程序提供的拍摄界面还可以包括一个拍照按键602(在前述图6和图7中未标出)，该拍照按键602的实质可以为拍摄界面中显示的一个功能控件。用户在完成如图7所示的构图之后，可以点击或触摸拍照按键602，手机可以响应于用户对拍照按键602的点击或触摸操作进行拍照，从而获取到待拍摄场景对应的照片。具体获取原理可以参考前述实施例中所述的手机实现拍摄功能的过程，不再赘述。
可选地，其他一些实施方式中，上述拍照按键602的功能也可以通过手机上的其他物理按键实现，在此不作限制。
由上所述，本申请实施例提供的该拍照方法能够在用户使用手机进行拍照时，结合当前的拍摄场景在手机提供的拍摄界面中显示构图辅助线，对用户进行构图引导。其中，构图辅助线可以为用户指示适合待拍摄场景的构图方式，引导用户按照构图辅助线指示的构图方式对手机的拍摄位置、角度等进行调整以完成构图，从而可以使得没有丰富拍摄经验的用户在进行拍照时也能够完成较好的构图，用户的体验可以更好。也即，本申请实施例可以使得用户在使用手机进行拍照时，能够根据云端推荐的构图辅助线，更轻松地知道通过哪种构图方式进行构图，不需要自己判断构图方式，也不要复杂的构图操作。
可选地，也有一些实施例中，上述S402中手机也可以不向云端发送完整的预览图像，而是只向云端发送预览图像的图像特征。例如，手机中可以预设有卷积神经网络(convolutional neural network，CNN)，手机在获取到预览图像后，可以通过CNN对 预览图像进行特征提取，得到预览图像的图像特征。然后，手机可以向云端发送预览图像的图像特征。相应地，云端可以根据预览图像的图像特征和场景信息，识别出预览图像中包含的元素，并记录预览图像中不同元素的位置和所占比例，不再赘述。
相对于手机直接向云端发送预览图像的方式而言，手机只向云端发送预览图像的图像特征，用于云端生成适合待拍摄场景的构图方式对应的构图辅助线，可以更好的保护用户隐私，防止用户隐私泄露。
可选地，本申请实施例中，对于云端根据预览图像中包含的元素，以及不同元素的位置和所占比例，结合预设的匹配规则中，确定适合待拍摄场景的构图方式对应的构图辅助线的方式，当云端无法根据预设的匹配规则，确定出适合待拍摄场景的构图方式对应的构图辅助线时(也即，匹配规则中不包含预览图像对应的拍摄场景时)，云端也可以不生成构图辅助线。相应地，手机也不会显示构图辅助线。或者，手机可以显示普通的九宫格辅助线。
一些实施例中，手机还具有开启和关闭构图辅助线显示的功能。例如，图9示出了本申请实施例提供的拍摄界面的又一示意图。如图9所示，手机的拍摄界面中还可以包括一个设置按键901，该设置按键901的实质也可以为拍摄界面中显示的一个功能控件。当用户点击或触摸设置按键901时，手机可以响应于用户对设置按键901的点击或触摸操作，将显示界面由拍摄界面切换至手机的拍照应用程序的设置界面。示例性地，图10示出了本申请实施例提供的设置界面的示意图。如图10所示，设置界面中至少包括：“构图辅助线”的文字标识，在文字标识一侧还设有一个构图辅助功能开关1001(实质为功能控件)。构图辅助功能开关1001由一个滑动区域和滑块(如图中黑色填充的区域即为滑块)组成。当用户需要开启构图辅助线的显示功能时，可以点击或触摸滑块，使得滑块移动至右侧(如图10中所示的位置)。手机可以响应于用户将滑块移动至右侧的操作，控制开启构图辅助线显示。当用户需要关闭构图辅助线的显示功能时，可以点击或触摸滑块，使得滑块移动至左侧。手机可以响应于用户将滑块移动至左侧的操作，控制关闭构图辅助线显示。
可以理解的，上述图9和图10所示的开启和关闭构图辅助线显示的功能中，手机可以只在构图辅助线显示功能开启时，将预览图像发送到云端以获取云端根据预览图像返回的构图辅助线。当关闭构图辅助线显示之后，手机在每次进行拍照时，拍摄界面中都不会显示构图辅助线。直至用户再次开启构图辅助线显示时，手机才会在拍摄界面中显示构图辅助线。
可选地，为提高用户的使用体验，另外一些实施例中，手机还可以在拍摄界面为用户提供用于关闭构图辅助线显示的功能按键。例如，图11示出了本申请实施例提供的拍摄界面的又一示意图。如图11所示，除预览画面、构图辅助线、以及按照按键外，手机提供的拍摄界面中还可以包括文字提示：“构图辅助线”1101，“构图辅助线”1101之后还包括一个功能控件：“X”1102。“构图辅助线”1101用于提示用户拍摄界面中显示的虚线为构图辅助线。功能控件“X”1102能够用于实现关闭构图辅助线显示的功能。如：当用户不喜欢或不需要构图辅助线进行构图引导时，用户可以点击或触摸功能控件“X”1102。手机可以响应于用户对功能控件“X”1102的点击或触摸 操作，不再显示构图辅助线。对于一些具有丰富拍摄经验的用户而言，可能更期待在拍照时按照自己的意愿去构图，不希望受到构图辅助线的干扰，那么通过这种方式可以进一步考虑更多用户的需求，提升用户的体验。
需要说明的是，虽然上述图11所示的拍摄界面中示例性地将“构图辅助线”1101和功能控件“X”1102显示在构图辅助线(图11中所示的虚线)的下方，但本申请对“构图辅助线”1101和功能控件“X”1102的显示区域均不作限制。例如，功能控件“X”1102还可以显示在构图辅助线上方、预览画面的右上角、拍摄界面中预览画面的一侧等。
可选地，另外一些实施例中，也可以不显示“构图辅助线”1101，只显示功能控件“X”1102。
或者，还有一些实施例中，用于实现关闭构图辅助线显示的功能的功能控件，在拍摄界面中也可以以其他形式的标识进行展现。例如，图12示出了本申请实施例提供的拍摄界面的又一示意图。如图12所示，除预览画面、构图辅助线、以及按照按键的功能控件外，手机提供的拍摄界面中还可以包括一个功能控件：“构图辅助”1201。当用户需要关闭构图辅助线的显示功能时，可以点击或触摸功能控件“构图辅助”1201。手机可以响应于用户对功能控件“构图辅助”1201的点击或触摸操作，控制关闭构图辅助线显示。可选地，当用户再次点击或触摸功能控件“构图辅助”1201时，手机还可以响应于用户对功能控件“构图辅助”1201的点击或触摸操作，控制开启构图辅助线显示。
又或者，还有一些实施例中，用于实现开启和关闭构图辅助线显示的功能的功能控件也可以是以其他物理按键(区别于拍摄界面中的虚拟按键)的形式实现。例如，可以对手机的拍照应用程序进行配置，使得手机上的某个物理按键(如：“音量+”或“音量-”)具有实现开启和关闭构图辅助线显示的功能。在手机启动运行拍照应用程序、显示预览画面和构图辅助线时，用户可以按压该物理按键，手机可以响应于用户对该物理按键的按压操作，不再显示构图辅助线或者开启显示构图辅助线。
应当理解，上述关于开启和关闭构图辅助线显示的功能的实现形式均为示例性说明，本申请在此不作限制。
可选地，本申请一些实施例中，云端根据预览图像和场景信息生成适合待拍摄场景的构图方式对应的构图辅助线时，同时还可以根据预览图像生成适合待拍摄场景的拍摄轮廓线。例如，云端根据预览图像中包含的元素、以及不同元素的位置和所占比例，生成适合待拍摄场景的构图方式对应的构图辅助线后，还可以进一步勾勒出预览图像中的元素(如：建筑、山体、道路等)的轮廓线，并根据适合待拍摄场景的构图方式对应的构图辅助线，将元素的轮廓线移动至能够满足该构图辅助线对应的构图方式的位置，得到拍摄轮廓线。换言之，拍摄轮廓线是根据构图辅助线和预览图像中的元素的轮廓线生成的。然后，云端可以同时向手机发送构图辅助线和拍摄轮廓线，手机可以在拍摄界面显示预览图像、构图辅助线、以及拍摄轮廓线。用户使用手机进行拍照前，参考拍摄界面中显示的构图辅助线，对手机的拍摄位置、角度等进行调整时，还可以进一步参考拍摄界面显示的拍摄轮廓线，将预览画面中的元素移动至拍摄轮廓 线所指示的位置。
例如，图13示出了本申请实施例提供的拍摄界面的又一示意图。假设待拍摄场景包含铁塔及其附近的风景时，按照前述实施例所述的方式，手机在拍摄界面显示的构图辅助线为图13中所示的三分构图对应的构图辅助线1301(图13中所示的两条竖线)，拍摄轮廓线可以为根据铁塔的轮廓线所生成的轮廓线1302。用户使用手机进行拍照前对拍摄画面进行预览时，可以参考图13所示的拍摄界面中显示的构图辅助线1301，将铁塔移动至两条竖线中的其中一条上(如：右侧竖线)，同时，参考拍摄轮廓线1302，将铁塔移动至拍摄轮廓线1302所在的区域内。通过这种方式，可以更进一步地简化用户进行构图的操作，提升用户体验。
可选地，与上述构图辅助线的实现方式类似，云端向手机返回的拍摄轮廓线也可以是预览图像中的多个像素点坐标组成的坐标数据集合(与构图辅助线的坐标数据集合不同)。
或者，云端也可以向手机返回一张包含构图辅助线和拍摄轮廓线的图像，该图像中，构图辅助线所在区域的像素点的像素值可以为0；拍摄轮廓线所在区域的像素点的像素值也可以为0。除构图辅助线和拍摄轮廓线之外的区域的像素点的像素值均可以为P，P为大于0、小于或等于255的整数。可选地，除构图辅助线之外的区域的像素点的像素值可以均为255。
需要说明的是，本申请对拍摄轮廓线所在区域的像素点的像素值也不作限制，拍摄轮廓线所在区域的像素点的像素值与除构图辅助线和拍摄轮廓线之外的区域的像素点的像素值不同即可，而拍摄轮廓线所在区域的像素点的像素值与构图辅助线所在区域的像素点的像素值可以相同，也可以不同。
可选地，本申请实施例提供的拍照方法中，手机响应于用户操作进行拍照，获取到待拍摄场景对应的照片之后，还可以将照片上传到云端。云端可以对照片进行构图优化，并将构图优化后的照片返回给手机进行显示，供用户选择是否保存。
例如，图14示出了本申请实施例提供的拍照方法的另一流程示意图。如图14所示，本申请实施例提供的拍照方法在前述图4所示的基础上，还可以包括S1401-S1404。
S1401、手机向云端发送对待拍摄场景进行拍照获得的初始照片。
可以理解的，通过前述图4所示的方法中的S407步骤，手机可以得到待拍摄场景对应的照片，该照片可以称为初始照片。
相应地，云端接收来自手机上传的初始照片。
S1402、云端对初始照片进行构图优化，得到至少一张构图优化后的照片。
S1403、云端向手机返回至少一张构图优化后的照片。
相应地，手机接收来自云端的构图优化后的照片。
S1404、手机显示初始照片和至少一张构图优化后的照片。
例如，手机在接收到构图优化后的照片后，可以显示构图优化后的照片、以及用户拍摄的初始照片，供用户选择其中的一张或多张进行保存。
下面对图14中所示的构图优化过程进行更具体的示例性说明。
示例性地，图15示出了本申请实施例中云端对初始照片进行构图优化的流程示意 图。如图15所示，云端至少可以包括：线条检测网络、图像矫正模块、显著性检测模块、美学评分网络。
手机将拍照获得的待拍摄场景对应的初始照片传到云端后，云端可以通过线条检测网络检测初始照片中包含的水平线、竖直线、建筑物的轮廓线等线条。线条检测网络检测得到初始照片中包含的线条后，图像矫正模块可以根据初始照片中包含的线条确定对初始照片进行矫正所需的变换矩阵，并采用该变换矩阵对初始照片进行矫正，得到矫正后的照片。图像矫正模块对初始照片进行矫正得到矫正后的照片后，显著性检测模块可以对矫正后的照片进行显著性检测，得到照片中的显著性结果。显著性检测模块得到矫正后的照片中的显著性结果后，云端可以根据美学评分网络以及前述矫正后的照片的显著性结果，对矫正后的照片进行裁切，得到多张候选裁切图，并对多张候选裁切图进行评分。然后，云端可以将多张候选裁切图中评分高的至少一张候选裁切图(如：评分排列最高的一张或两张)作为构图优化后的照片发送给手机。
示例性地，此处所述的线条检测网络、以及图像矫正模块可以与前述图2A中所示的线条检测网络、以及图像矫正模块相同。线条检测网络的具体组成可以参考前述图2B所示。
例如，云端通过线条检测网络检测初始照片中包含的水平线、竖直线、建筑物的轮廓线等线条的过程可以包括：将初始照片输入主干网络后，主干网络提取初始照片中的特征，输出初始照片对应的共享卷积特征图至连接点预测模块。连接点预测模块根据共享卷积特征图输出初始照片对应的候选连接点，并传输给线段采样模块。线段采样模块根据候选连接点预测出初始照片包含的线条(或称为线段)。线段校正模块对预测出的线条进行分类，最终输出检测到的初始照片中包含的线条。
图像矫正模块根据初始照片中包含的线条确定对初始照片进行矫正的效果，可以参考前述图2C和图2D所示的对预览图像进行矫正的效果，矫正原理相同，不再赘述。
示例性地，显著性检测模块可以采用Mask RCNN分割方法分割出矫正后的照片中的人体、动物、建筑等关注点作为显著性结果。
可选地，对于没有明显显著性的照片，如：照片中不包含人体、动物、建筑等可以作为显著性结果的关注点时，显著性检测模块可以分割出矫正后的照片中与美学构图相关的元素作为显著性结果，如，与美学构图相关的元素可以是分界线、道路、桥梁等。
需要说明的是，本申请实施例中，哪些元素可以作为显著性结果、以及哪些元素可以是与美学构图相关的元素，可以是人为定义的。也即，本申请实施例中，显著性检测模块可以根据人为定义的显著性元素，分割照片中的显著性结果，在此不作限制。
示例性地，美学评分网络至少可以包括：训练好的目标检测(single shot multiBox detector，)SSD网络、回归模型。
云端可以将矫正后的照片、以及矫正后的照片的显著性结果输入SSD网络，通过SSD网络计算当矫正后的照片按照预设的多个裁切框(可以称为第一裁切框)分别进行裁切时每个裁切框对应的评分。其中，预设的裁切框数量不作限制，可以人为定义或配置。然后，云端可以根据SSD网络输出的多个裁切框分别对应的评分，对预设的 多个裁切框进行排序，确定出评分最高的裁切框。之后，云端可以将矫正后的照片、矫正后的照片的显著性结果、以及通过SSD网络得到的前述评分最高的裁切框输入回归模型进行回归，得到多个最终裁切框(可以称为第二裁切框)以及对应的评分。云端根据多个最终裁切框对矫正后的照片进行裁切即可得到多张候选裁切图，每张候选裁切图对应的评分即为最终裁切框对应的评分。
例如，图16示出了本申请实施例提供的回归原理示意图。如图16所示，假设矫正后的照片的大小为m1*n1(即，包含m1*n1个像素点，如图16中每个小方格可以看作一个像素点)。通过SSD网络得到的评分最高的裁切框为图16中粗实线所示的矩形框。该矩形框的左上角像素点a的坐标为(x1，y1)，右下角像素点b的坐标为(x2，y2)。将像素点a与矫正后的照片的左上角的像素点连接时，可以将连线作为对角线，在矫正后的照片中划分出一个矩形区域(以下称为第一区域)，也即，第一区域的左上角为矫正后的照片的左上角，右下角为像素点a。将像素点b与矫正后的照片的右下角的像素点连接时，也可以将连线作为对角线，在矫正后的照片中划分出一个矩形区域(以下称为第二区域)，也即，第二区域的左上角为像素点b，右下角为矫正后的照片的右下角。回归模型在对评分最高的裁切框进行回归时，可以将裁切框的左上角限制在前述第一区域内、右下角限制在前述第二区域内进行回归，例如，回归得到的其他的最终裁切框可以如图16中的多个虚线框所示。示例性地，第一区域和第二区域的大小相同，为m2*n2。可以理解，m2小于m1，n2小于n1，且m1、m2、n1、n2均为正整数。
可选地，可以采用一些公开的数据集训练获取上述SSD网络，如：对比照片组成数据集(comparative photo composition dataset，CPC)。CPC数据集中包括多张照片(或称为图片)，每张照片标注有采用不同的裁切方式时对应的美学打分。在训练SSD网络时，可以先利用Mask RCNN对CPC数据集中的照片进行分割，得到每张照片对应的显著性结果。然后，将每张照片和对应的显著性结果输入SSD网络进行训练，使得训练好的SSD网络能够给不同尺度的裁切框进行打分。
对于上述矫正后的图片对应的多张候选裁切图，云端可以将多张候选裁切图中评分高的至少一张候选裁切图(如：评分排列最高的一张或两张)作为构图优化后的照片发送给手机。手机在接收到来自云端的构图优化照片后，可以显示初始照片和构图优化后的照片，供用户选择是否保存。如，用户可以选择第一照片进行保存，第一照片可以是初始照片、以及构图优化后的照片中的一张或多张。
举例说明，图17示出了本申请实施例提供的构图优化的示意图。假设手机拍照获得的待拍摄场景对应的初始照片如图17中的(a)所示，则云端对初始照片进行图像矫正后的照片可以如图17中的(b)所示。云端对如图17中的(b)所示的照片进行显著性检测得到的显著性结果可以如图17中的(c)所示，包含骑马的人、灯塔。之后，云端可以将如图17中的(b)所示的照片、以及如图17中的(c)所示的显著性结果输入美学评分网络，得到美学评分网络输出的多张候选裁切图，并从多张候选裁切图中选择评分最高的两张作为构图优化后的照片。示例性地，构图优化后的照片可以如图17中的(d)和(e)所示。
图18示出了本申请实施例提供的照片显示界面的示意图。可选地，手机在用户进行拍照操作后，可以将图17中的(a)所示的初始照片通过如图18所示的照片显示界面进行显示，供用户查看拍照效果。例如，该照片显示界面可以为手机的图库的播放界面。另外，请继续参考图18所示，手机还可以将接收到的如图17中的(d)和(e)所示的两张构图优化后的照片显示在初始照片的下方。其中，初始照片的上方还包括一个选择功能控件“A”1801，两张构图优化后的照片的上方分别包括选择功能控件“B1”1802和选择功能控件“B2”1803。照片显示界面中还包括确定按键1804(图18中所示的对号)和取消按键1805(图18中所示的叉号)，确定按键1804和取消按键1805均为功能控件，功能不同。
当用户对功能控件“A”1801、功能控件“B1”1802、以及功能控件“B2”1803中的某一个或某几个功能控件进行点击或触摸时，手机可以响应于用户对相应功能控件的点击或触摸操作，选中对应的照片。然后，当用户再点击或触摸下方的确定按键1804时，手机可以响应于用户对确定按键1804的点击或触摸操作，将前述选中的照片进行保存。例如，当用户想要保存功能控件“B2”1803对应的照片时，可以点击选择功能控件“B2”1803，选中该照片，然后，点击下方的确定按键1804，从而保存该照片。
可选地，当用户对每张照片均不满意，想要重新拍照时，可以点击或触摸下方的取消按键1805。手机可以响应于用户对取消按键1805的点击或触摸操作，重新切换至拍摄界面供用户进行拍照。这时，图18所示的手机界面中显示的照片均不会被保存，或者，手机可以只保存初始照片。
本申请实施例中，云端通过对手机拍摄的初始照片进行进一步的构图优化，并返回给手机构图优化后的照片，可以提高构图成功率，构图优化后的照片不会出现旋转、透视等畸变，能够为用户提供更好的照片选择。另外，手机同时显示用户拍摄的初始照片和构图优化后的照片，可以考虑到用户的需求，使得用户既可以选择自己拍摄的照片，也可以选择云端推荐的构图优化后的照片。
可选地，一些实施例中，手机在执行S1401之前，可以先检测初始照片是否为人像照片，如果初始照片是人像照片，则手机可以不执行S1401的步骤，直接保存人像照片。也即，手机不向云端发送初始照片，云端不对初始照片进行后续的构图优化。如果初始照片不是人像照片，则手机可以按照图14所示的流程，继续执行S1401的步骤，向云端发送初始照片，云端对初始照片进行后续的构图优化。
手机检测到初始照片是人像照片时，不向云端发送初始照片对其进行构图优化，可以更好的保护用户隐私，防止用户隐私泄露。
可选地，图22示出了本申请实施例提供的拍照方法的另一流程示意图。如图22所示，本申请实施例提供的拍照方法在前述图14所示的基础上，还可以包括步骤S2201至S2203。
S2201、手机响应于用户的选择操作，选择第一照片。
所述第一照片包括初始照片、以及至少一张构图优化后的照片中的一张或多张。
如前文所述，并且如图18所示，手机同时显示用户拍摄的初始照片和至少一张构 图优化后的照片，用户选择对构图满意的照片进行保存。可以理解，用户既可以选择自己拍摄的照片，也可以选择云端推荐的构图优化后的照片。可选地，用户可以选择初始照片和至少一张构图优化后的照片中的一张照片进行保存，也可以选择多张照片进行保存。上述保存的一张或多张照片，即为第一照片。
S2202、手机向云端发送所述第一照片。
相应地，云端接收手机发来的所述第一照片。
S2203、云端可以根据第一照片，获取用户的美学偏好。
下面对获取用户的美学偏好进行更具体地示例性说明。
为了获取用户的美学偏好，云端可以根据第一照片训练云端的美学评分网络。如前文所述，美学评分网络至少可以包括目标检测(single shot multiBox detector，SSD)网络、回归模型。
在一个示例中，手机将用户选择的第一照片上传到云端之后，云端通过显著性检测模块检测第一照片的显著性，生成每张第一照片的显著性结果。然后，云端可以将每张第一照片的显著性结果作为输入，第一照片的构图方式作为输出对美学评分网络进行训练。训练好的美学评分网络可以学习到第一照片中包含的元素、以及不同元素的位置和所占比例，与第一照片的构图方式的映射关系。从而使云端获取用户的美学偏好，生成适合待拍摄场景的构图方式。示例性地，云端对美学评分网络进行训练，可以是云端对SSD网络进行训练，即，云端将每张第一照片对应的显著性结果输入SSD网络进行训练，使得训练好的SSD网络能够给不同尺度的裁切框进行打分，可以针对待拍摄场景的构图方式给出排序。
在另一个示例中，手机将用户选择的第一照片上传到云端之后，云端通过显著性检测模块检测第一照片的显著性，生成每张第一照片的显著性结果。然后，云端可以将每张第一照片的显著性结果、对应拍摄场景的场景信息作为输入，第一照片的构图方式作为输出对美学评分网络进行训练。训练好的美学评分网络可以学习到第一照片对应的拍摄场景的场景信息、第一照片中包含的元素、以及不同元素的位置和所占比例，与第一照片的构图方式的映射关系。从而使云端获取用户的美学偏好，生成适合待拍摄场景的构图方式。示例性地，云端对美学评分网络进行训练，可以是云端对SSD网络进行训练，即，云端将每张第一照片对应的显著性结果输入SSD网络进行训练，使得训练好的SSD网络能够给不同尺度的裁切框进行打分，可以针对待拍摄场景的构图方式给出排序。可以理解，第一照片对应拍摄场景的场景信息，可以是在步骤S401中已经由手机上传到云端的场景信息，云端可以根据第一照片与上述场景信息的对应关系，获取存储在云端的与所述第一照片对应的场景信息。可选地，手机向云端发送第一照片时，可以将对应拍摄场景的场景信息一并发送到云端。云端完成美学评分网络的训练后，可以将训练好的美学评分网络迁移到与该用户对应的云端存储空间中，从而获取用户的美学偏好。可选地，当云端已经存储了与用户美学偏好相应的用户定制的美学评分网络时，云端可以将训练好的美学评分网络替换已有的用户定制的美学评分网络。
在上述实施例中，终端设备将用户的拍照构图选择结果上传至云端，云端对美学 评分网络进行训练，获取用户的美学偏好。通过端云协同的拍照构图私人定制方法，可以依靠云端的强算力，有效降低终端设备硬件要求，持续推荐更加符合用户习惯的拍照构图，减少用户的学习成本，提升用户拍照构图体验。
可选地，在一些实施例中，手机也可以不向云端发送完整的第一照片，而是向云端发送第一照片的图像特征。例如，手机中可以预设有卷积神经网络(convolutional neural network，CNN)，手机在获取到第一照片后，可以通过CNN对第一照片进行特征提取，得到第一照片的图像特征。然后，手机可以向云端发送第一照片的图像特征。相应地，云端可以根据第一照片的图像特征和场景信息，识别出第一照片中包含的元素，并记录第一照片中不同元素的位置和所占比例，不再赘述。
相对于手机直接向云端发送第一照片的方式，手机只向云端发送第一照片的图像特征，用于云端获取用户的美学偏好，可以更好的保护用户隐私，防止用户隐私泄露。
在另一些实施例中，手机可以向云端发送第一照片的标识信息。第一照片的标识信息可以是第一照片的文件名，也可以是系统为第一照片分配的唯一标识符。所述标识信息还可以是基于第一照片计算的照片哈希值。所述哈希值是云端针对前述待拍摄场景对应的照片、至少一张构图优化后的照片计算的哈希值，可以用于唯一地标记照片。可以理解，计算哈希值可以采用本领域已有的哈希值算法，此处不再赘述。云端接收到手机发来地第一照片的标识信息后，可以根据第一照片的标识信息查询存储在云端的对应的第一照片，以便云端根据该对应的第一照片获取用户的美学偏好。
相对于手机直接向云端发送第一照片的方式，手机向云端发送第一照片的标识信息，用于云端获取用户的美学偏好，可以更好的保护用户隐私，防止用户隐私泄露，且可以减少手机与云端的通信数据量，提高手机传输数据的效率。
最后，还需要说明的是，以上实施例中是以手机按照第二频率向云端发送获取到的预览图像和场景信息为例进行说明的。但应当理解，本申请一些实现场景中，手机向云端发送场景信息的频率也可以与向云端发送预览图像的频率不同。例如，手机可以按照第三频率向云端发送获取到的预览图像、按照第四频率向云端发送获取到的场景信息，第三频率可以小于或大于第四频率，在此不作限制。
图23示出了本申请实施例提供的拍照方法的另一流程示意图。如图23所示，本申请实施例提供的拍照方法包括步骤S2301-S2308。
S2301、手机获取待编辑照片和待编辑照片对应的场景信息。
在一些实施例中，待编辑照片可以是手机拍照获取的照片，待编辑照片对应的场景信息可以是手机在拍照时采集得到的场景信息。用户使用手机进行拍照后，手机可以在编辑界面显示拍摄照片的缩略图，以便用户浏览和/或编辑拍摄照片。上述拍照获取待编辑照片以及对应的场景信息，可以参照前文所述的拍照过程和采集场景信息过程，此处不再赘述。
在另一些实施例中，待编辑照片可以是用户在手机的相册APP中浏览并选择感兴趣的照片，待编辑照片对应的场景信息可以是拍摄待编辑照片时手机采集的对应于拍摄场景的场景信息。用户选择上述照片后，手机可以在编辑界面显示该照片的缩略图。上述待编辑照片和场景信息存储在手机中。其中，场景信息可以与待编辑照片一同保 存，也可以与待编辑照片分开保存。在一个示例中，场景信息可以存储在待编辑照片的电子文件中，例如RAW、JPG或IMG等格式的图像文件，本申请对此不作限制。在选择待编辑照片后，手机从该待编辑照片中提取场景信息。可以理解，手机也可以将包括场景信息的待编辑照片发送给云端，由云端从待编辑照片中提取场景信息。在另一个示例中，场景信息可以存储在待编辑照片对应的照片属性文件中，例如可扩展标记语言(Extensible Markup Language，XML)格式文件。手机向云端发送场景信息时，可以将照片属性文件发送给云端。可以理解，手机还可以采取其它已有的方式存储场景信息，本申请对此不作限制。
S2302、手机向云端发送待编辑照片和待编辑照片对应的场景信息。
可以理解，通过前述步骤S2301，手机可以获取待编辑照片和待编辑照片对应的场景信息，并将该待编辑照片和待编辑照片对应的场景信息发送到云端。
相应地，云端接收来自手机上传的待编辑照片和对应的场景信息。
可选地，手机可以响应于用户的编辑操作，向云端发送待编辑照片和待编辑照片对应的场景信息。用户针对待编辑照片进行编辑操作，可以在手机的照片编辑界面中进行。示例性地，图24示出了本申请实施例提供的照片编辑界面的一种示意图。如图24所示，该照片编辑界面显示待编辑照片的预览图，待编辑照片的预览图的下方可以显示文字提示2401、构图优化功能控件2402和2403。其中，文字提示2401可以显示例如构图优化的字样，构图优化功能控件2402和2403分别用于取消操作和确认操作。当用户对构图优化控件2403点击或触摸时，手机可以响应于用户的点击或触摸操作，启动构图优化功能，向云端发送待编辑照片和待编辑照片对应的场景信息。相应地，用户对构图优化功能控件2402点击或触摸时，手机可以不启用构图优化功能。可以理解，上述编辑界面、文字提示以及功能控件还可以采用其它已有的显示风格或排列方式，本申请对此不作限制。
S2303、云端对待编辑照片进行构图优化，得到至少一张构图优化后的照片。
示例性地，与图15所示的流程相似，云端至少可以包括：线条检测网络、图像矫正模块、显著性检测模块、美学评分网络。其中，云端采用线条检测网络、图像矫正模块和显著性检测模块对待编辑照片进行的处理过程，可以参照前文所述的内容进行，此处不再赘述。云端可以将矫正后的待编辑照片、矫正后的照片的显著性结果以及场景信息输入美学评分网络，以获得一张或多张候选裁切图作为构图优化后的照片。
示例性地，美学评分网络至少可以包括：训练好的SSD网络、回归模型。
云端可以将矫正后的照片、矫正后的照片的显著性结果以及场景信息输入SSD网络，通过SSD网络计算当矫正后的照片按照预设的多个裁切框(可以称为第一裁切框)分别进行裁切时每个裁切框对应的评分。其中，预设的裁切框数量不作限制，可以人为定义或配置。然后，云端可以根据SSD网络输出的多个裁切框分别对应的评分，对预设的多个裁切框进行排序，确定出评分最高的裁切框。之后，云端可以将矫正后的照片、矫正后的照片的显著性结果、以及通过SSD网络得到的前述评分最高的裁切框输入回归模型进行回归，得到多个最终裁切框(可以称为第二裁切框)以及对应的评分。云端根据多个最终裁切框对矫正后的照片进行裁切即可得到多张候选裁切图，每 张候选裁切图对应的评分即为最终裁切框对应的评分。可选地，回归模型可以采用如前文所述的回归模型，此处不再赘述。
可选地，可以采用一些公开的数据集训练获取上述SSD网络，如：对比照片组成数据集(comparative photo composition dataset，CPC)。CPC数据集中包括多张照片(或称为图片)，每张照片标注有采用不同的裁切方式时对应的美学打分。在训练SSD网络时，可以先利用Mask RCNN对CPC数据集中的照片进行分割，得到每张照片对应的显著性结果。然后，将每张照片、照片对应的显著性结果以及场景信息输入SSD网络进行训练，使得训练好的SSD网络能够给不同尺度的裁切框进行打分。
示例性地，云端的美学评分网络可以是通用美学评分网络，即该美学评分网络可以供任何用户使用。可选地，云端的美学评分网络也可以是与上传待编辑照片的用户的对应的用户定制美学评分网络，该用户定制美学评分网络可以存储在与该用户对应的云端存储空间中。在一些实施例中，当云端仅存储有通用美学评分网络时，例如用户首次使用构图优化功能，则云端采用通用美学评分网络对待编辑照片进行构图优化，得到至少一张构图优化后的照片。在另一些实施例中，当云端存储有用户定制美学评分网络时，则云端采用该用户定制美学评分网络对待编辑照片进行构图优化，得到至少一张构图优化后的照片。
S2304、云端向手机返回至少一张构图优化后的照片。
相应地，手机接收来自云端的至少一张构图优化后的照片。
S2305、手机显示初始照片和至少一张构图优化后的照片。
例如，手机在接收到构图优化后的照片后，可以显示构图优化后的照片、以及用户拍摄的初始照片，供用户选择其中的一张或多张进行保存。
S2306、手机响应于用户的保存操作，保存第一照片。
所述第一照片包括初始照片、以及至少一张构图优化后的照片中的一张或多张。
S2307、手机向云端发送所述保存的第一照片。
相应地，云端接收手机上传的第一照片。
S2308、云端可以根据保存的第一照片，获取用户的美学偏好。
上述步骤S2304至S2308分别与前述步骤S1403、S1404、S2201至S2203的过程相同，此处不再赘述。需要说明的是，在步骤S2308中，在云端已经存储有用户的美学偏好的情况下，云端获取用户的美学偏好之后，可以更新已有的用户的美学偏好。本实施例中，终端设备将用户拍摄的照片或已有照片上传至云端，云端基于美学评分网络向终端设备返回构图优化后的照片，终端设备再将用户的拍照构图选择结果上传至云端，云端对美学评分网络进行训练，获取用户的美学偏好。这种端云协同的拍照构图私人定制方法，可以依靠云端的强算力，有效降低终端设备硬件要求，持续推荐更加符合用户习惯的拍照构图，减少用户学习成本，提升用户拍照构图体验。
在一些实施例中，终端设备在向云端发送第一照片之前，可以先检测第一照片是否为人像照片，如果第一照片是人像照片，则终端设备可以直接保存人像照片，不向云端发送第一照片，以更好的保护用户隐私，防止用户隐私泄露。
可选地，在一些实施例中，手机也可以不向云端发送完整的第一照片，而是只向 云端发送第一照片的图像特征。上述过程可以参照前述的手机向云端发送第一照片的图像特征的过程，此处不再赘述。相对于手机直接向云端发送第一照片的方式而言，手机只向云端发送第一照片的图像特征，用于云端获取用户的美学偏好，可以更好的保护用户隐私，防止用户隐私泄露。
在另一些实施例中，手机可以向云端发送第一照片的标识信息。上述过程可以参照前述的手机向云端发送第一照片的标识信息的过程，此处不再赘述。相对于手机直接向云端发送第一照片的方式，手机向云端发送第一照片的标识信息，用于云端获取用户的美学偏好，可以更好的保护用户隐私，防止用户隐私泄露，且可以减少手机与云端的通信数据量，提高手机传输数据的效率。
图25示出了本申请实施例提供的拍照方法的另一流程示意图。如图25所示，本申请实施例提供的拍照方法包括步骤S2501-S2507。
S2501、手机获取待编辑照片。
在一些实施例中，待编辑照片可以是手机拍照获取的照片。用户使用手机进行拍照后，手机可以在编辑界面显示拍摄照片的缩略图，以便用户浏览和/或编辑拍摄照片。上述拍照获取待编辑照片，可以参照前文所述的拍照过程，此处不再赘述。
在另一些实施例中，待编辑照片可以是用户在手机的相册APP中浏览照片的缩略图，然后感兴趣的一幅照片作为待编辑照片。待编辑照片可以存储在手机中，也可以是存储在云端或服务器上。用户选择上述照片后，手机可以在编辑界面显示该照片的缩略图。
S2502、手机响应于用户对待编辑照片的编辑操作，获取编辑后的照片。
用户的编辑操作可以在手机的照片编辑界面中进行。在一些实施例中，手机响应于用户对待编辑照片的编辑操作，可以根据本地保存的基础画质算法获取中间照片。用户可以进一步对中间照片进行手动编辑，调整或设置照片参数，从而获得符合用户美学偏好的编辑后的照片。基础画质算法可以包括美颜算法、照片风格算法、滤镜算法、曝光算法、降噪算法等。通过基础画质算法，手机可以提供多种功能，例如磨皮、瘦脸、大眼，也包括瘦鼻、长鼻、下巴、额头，还包括开眼角、嘴型、微笑嘴角等功能。在一个示例中，当针对待编辑照片应用美颜算法时，可以对照片中的肖像进行自动美颜，例如增白肖像的皮肤，对皮肤进行磨皮等。在另一个示例中，当针对待编辑照片应用自动曝光算法时，可以对照片进行自动曝光，例如调整照片的亮度、对比度等参数，从而更好更清晰地呈现照片的内容。在其它一些示例中，当针对待编辑照片应用滤镜算法时，可以对照片施加例如老照片、电影风格、黑白等滤镜效果。
可以理解，对待编辑照片应用基础画质算法，是对照片的全部区域(即整张照片)或局部区域的照片参数进行调整或设置。照片参数可以包括亮度、对比度、饱和度、锐度、清晰度、噪点、高亮(或称“亮部”或“高光”)、阴影(或称“暗部”)或者色调等参数。照片参数可以存储在照片文件中，也可以存储在照片文件对应的照片属性文件中。所述照片属性文件可以参照前文描述，此处不再赘述。每个基础画质算法包括对应的基础画质算法参数，该基础画质算法参数用于确定上述照片参数应用在照片上的程度或强度。基于相应的基础画质算法参数，可以实现照片的例如美颜、照 片风格、滤镜、曝光、降噪等编辑操作。
为了更好地说明上述编辑过程，下面结合附图进一步说明。图26A至图26C示出了本申请实施例提供的一种照片编辑界面的另一种示意图。如图26A所示，该照片编辑界面显示待编辑照片的预览图，预览图的下方可以显示功能控件，例如美颜控件2601、滤镜控件2602、一键优化控件2603、更多控件2604等。在图26A中，待编辑照片例如为人像，其面部存在一些皱纹和斑点。用户可以点击美颜控件2601对待编辑照片进行美颜。如图26B所示，手机对待编辑照片进行美颜，并将美颜后的中间照片的预览图显示在照片编辑界面。可以理解，美颜算法针对待编辑照片应用了的一组照片参数集合，例如亮度、对比度、色调、消除杂色、去除斑纹等。预览图的下方显示美颜强度控制按钮2605，当手机自动完成美颜时，控制按钮2605处于参数值为5的中间位置。如前所述，该过程体现了基础画质算法参数对待编辑照片的影响。由于用户的美学偏好存在不同，用户可能认为美颜程度太弱或者太强，则用户可以通过进一步调整控制按钮2605，以减弱或增强美颜的程度。例如，在图26B中，中间照片仍然存在皱纹(斑点已经被美颜去除)。用户可以向右拖动控制按钮2605，例如调整到如图26C所示的参数值7，从而消除了人像照片中的皱纹和斑点。编辑后的照片的预览图显示在编辑界面中。
可选地，图27A至图27C示出了本申请实施例提供的照片编辑界面的又一种示意图。如图27A所示，该照片编辑界面显示待编辑照片的预览图，预览图的下方可以显示功能控件，例如美颜控件2701、滤镜控件2702、一键优化控件2703、更多控件2704等。在图27A中，待编辑照片例如为风景，照片的亮度较暗，对比度也偏弱。用户可以点击一键优化控件2703对待编辑照片进行优化。如图27B所示，手机对待编辑照片进行优化，并将优化后的中间照片预览图显示在照片编辑界面。该优化过程应用了一组照片参数集合，例如亮度、对比度、饱和度、锐度等。预览图的下方分别显示亮度控件2705、对比度控件2706、饱和度控件2707和锐度控件2708。用户可以通过上述控件调整或设置对应的亮度、对比度、饱和度和锐度值。以亮度控件2601为例，用户向左拖动控件时，亮度值减小，照片亮度降低；用户向右拖动控件时，亮度值增大，照片亮度提高。在手机自动完成优化的情况下，上述控件均处于参数值为0的中间位置。用户根据美学偏好对中间照片进行调整。例如，在图27B中，中间照片仍然偏暗，对比度也较弱。用户可以拖动上述控件调整或设置相应参数对照片进行编辑。如图27C所示，用户将亮度值调整为1，对比度值调整为2，饱和度值保持不变，锐度值调整为1，得到编辑后的照片。编辑后的照片的预览图显示在照片编辑界面中。
在其它一些实施例中，在手机根据本地保存的基础画质算法获取中间照片之后，如果用户对该中间照片满意，可以不对该中间照片进行编辑，手机将该中间照片作为编辑后的照片。换句话说，如果中间照片符合用户的审美偏好，手机可以将该中间照片作为训练云端的拍照风格优化网络的输入数据，以便获取用户的美学偏好。
可以理解，在上述实施例中，编辑界面、功能控件还可以采用其它已有的显示风格或排列方式，以实现对照片的相应调整，本申请对此不作限制。
S2503、手机向云端发送待编辑照片和编辑后的照片。
相应地，云端接收来自手机上传的待编辑照片和编辑后的照片。
S2504、云端根据待编辑照片和编辑后的照片，获取用户的美学偏好。
S2505、云端根据用户的美学偏好获取基础画质优化参数。
S2506、云端向手机返回基础画质优化参数。
相应地，手机接收来自云端的基础画质优化参数。
S2507、手机根据基础画质优化参数更新本地的基础画质算法。
示例性地，手机接收到云端发送的基础画质优化参数之后，可以替换手机本地存储的基础画质参数，即，手机通过基础画质优化参数更新本地的基础画质算法。在完成基础画质算法更新之后，当手机通过基础画质算法处理照片时，可以获取更加符合用户习惯的拍照风格的照片，提升用户拍照体验。
下面对图25中所示的获取用户的美学偏好进行更具体地示例性说明。
示例性地，云端可以对基础画质算法进行建模，得到基础画质算法参数与成像效果的关系函数(模型)。基础画质算法参数可以基于现有的基础画质算法初步确定。在一个示例中，对于自动曝光算法，基础画质算法参数可以基于现有的平均亮度法初步确定，云端可以建立自动曝光算法参数与亮度成像效果的关系函数。在另一个示例中，对于降噪算法，基础画质算法参数可以基于现有的高斯去噪法初步确定，云端可以建立降噪算法参数与降噪成像效果的关系函数。在又一个示例中，针对美颜算法，例如基于双边滤波法对肤色区域做平滑处理，然后针对肤色调节色调，云端可以建立美颜算法参数与美颜成像效果的关系函数。可以理解，基础画质算法还包括饱和度算法、色调调整算法等，本申请对此不作限制。
为了获得用户的美学偏好，云端可以根据待编辑照片和编辑后的照片，对美学评分网络进行训练。具体地，云端可以对前述的基础画质算法参数与成像效果的关系函数进行训练，并将训练好的关系函数迁移到与用户对应的云端存储空间中，以获取用户的美学偏好。示例性地，云端对关系函数进行训练，可以将待编辑照片作为输入，编辑后的照片作为输出对关系函数进行训练。基于训练好的关系函数，云端可以获得优化的基础画质算法参数(也称“基础画质优化参数”)，例如优化的自动曝光算法参数、降噪算法参数、美颜算法参数等。需要说明的是，在云端已经存储有用户的美学偏好的情况下，云端可以更新已有的用户美学偏好，即，云端将训练好的关系函数替换之前的关系函数。
本实施例中，终端设备将用户编辑前后的照片上传到云端，对云端部署的拍照风格优化网络进行训练，迁移得到用户定制的拍照风格优化网络，并同步更新终端设备的基础画质算法。这种端云协同的拍照风格私人定制方法，可以依靠云端的强算力，有效降低终端设备硬件要求，利用云端对基础画质算法的建模及更新，持续推荐更加符合用户习惯的拍照风格，提升用户拍照体验。
可选地，在一些实施例中，上述步骤S2502中手机也可以不向云端发送完整的待编辑照片和编辑后的照片，而是只向云端发送待编辑照片和编辑后的照片的图像特征。例如，手机中可以预设有CNN网络，手机在获取到待编辑照片和编辑后的照片后，可以通过CNN网络分别对待编辑照片和编辑后的照片进行特征提取，得到待编辑照片和 编辑后的照片的图像特征。然后，手机可以向云端发送待编辑照片和编辑后的照片的图像特征。相应地，云端可以根据待编辑照片和编辑后的照片的图像特征，对美学评分网络进行训练，不再赘述。
相对于手机直接向云端发送待编辑照片和编辑后的照片的方式而言，手机只向云端发送待编辑照片和编辑后的照片的图像特征，用于云端获取用户定制的拍照风格优化网络，可以更好的保护用户隐私，防止用户隐私泄露。
对应于前述实施例中所述的拍照方法，本申请实施例还提供一种端云协同系统。该端云协同系统可以包括：终端设备和云端。终端设备通过无线网络与云端连接。终端设备与云端配合实现如前述实施例中所述的拍照方法。
该端云协同系统具体可以参考前述实施例中的图2中所示，此处不再赘述。其中，终端设备具有拍照功能。
本申请实施例还提供一种终端设备，该终端设备可以包括拍照装置。该拍照装置可以用于实现前述实施例中所述的拍照方法中终端设备所执行的功能。该拍照装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元。
例如，图19示出了本申请实施例提供的拍照装置的结构示意图。如图19所示，该拍照装置可以包括：相机模块1901，显示模块1902，传感器模块1903，发送模块1904，接收模块1905等。
其中，相机模块1901，用于响应于用户操作，获取待拍摄场景对应的预览图像；显示模块1902，用于在拍摄界面显示所述相机模块1901获取的预览图像；传感器模块1903，用于采集所述待拍摄场景的场景信息；发送模块1904，用于向云端发送所述相机模块1901获取到的预览图像、以及所述传感器模块1903获取到的场景信息；接收模块1905，用于接收来自所述云端的构图辅助线，所述构图辅助线用于指示适合所述待拍摄场景的构图方式；所述显示模块1902，还用于将所述构图辅助线与预览图像显示在所述拍摄界面；所述相机模块1901，还用于接收到用户的拍照操作后，响应于用户的拍照操作，获取所述待拍摄场景对应的照片。
可选地，发送模块1904向云端发送场景信息的频率小于向云端发送预览图像的频率。
可选地，显示模块1902，还用于响应于用户关闭构图辅助线的显示功能的操作，不显示构图辅助线。
可选地，发送模块1904，具体用于通过预设的卷积神经网络提取获取到的预览图像的图像特征；向云端发送获取到的预览图像的图像特征。
或者，通过预设的卷积神经网络提取获取到的预览图像的图像特征的功能，也可以由单独的一个特征提取模块(图中未示出)完成，如：该装置还包括特征提取模块。
可选地，接收模块1905，还用于接收来自云端的拍摄轮廓线；显示模块1902，还用于将拍摄轮廓线、构图辅助线与正在显示的预览图像一同显示在拍摄界面。
可选地，发送模块1904，还用于向云端发送待拍摄场景对应的照片；接收模块1905，还用于接收来自云端的所述照片对应的至少一张构图优化后的照片；显示模块1902， 还用于显示所述照片、以及所述至少一张构图优化后的照片，并响应于用户对第一照片的保存操作，保存第一照片，第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张。可选地，发送模块1904，还用于向云端发送第一照片，所述第一照片用于所述云端获取用户的美学偏好。
可选地，发送模块1904，还用于向云端发送第一照片的图像特征。
本申请实施例还提供一种云端服务器，该云端服务器可以用于实现前述实施例中所述的拍照方法中云端所执行的功能。该拍照装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元。
例如，图20示出了本申请实施例提供的电子设备的另一结构示意图。如图20所示，该电子设备可以包括：接收模块2001，构图模块2002，发送模块2003等。
其中，接收模块2001，用于接收来自终端设备的待拍摄场景对应的预览图像和场景信息；构图模块2002，用于根据所述预览图像和所述场景信息，生成适合所述待拍摄场景的构图方式对应的构图辅助线；发送模块2003，用于向所述终端设备发送所述构图辅助线。
一种实施方式中，构图模块2002，具体用于根据预览图像和场景信息，识别预览图像中包含的元素，并记录预览图像中不同元素的位置和所占比例；根据预览图像中包含的元素，以及不同元素的位置和所占比例，按照预设的匹配规则确定与待拍摄场景匹配的构图方式对应的构图辅助线。
其中，匹配规则包括至少一种类型的待拍摄场景与构图辅助线之间的对应关系，不同类型的待拍摄场景中包含的元素、以及不同元素的位置和所占比例不同。
上述匹配规则可以是人为定义的规则。
示例性地，预览图像中包含的元素可以是天空、海水、草地、人物等。元素的位置是指预览图像中元素所在区域的像素点坐标，元素所占的比例是指预览图像中元素所在区域的像素点数量占整个预览图像的像素点数量的比值。
可选地，构图模块2002，具体用于采用第一方法对预览图像进行分割，然后基于分割结果、结合场景信息识别预览图像中包含的元素。
其中，场景信息用于辅助构图模块2002基于分割结果快速识别预览图像中包含的元素。
例如，当场景信息包括的位置信息为海边时，可以表示预览图像中可能含有海水，能够辅助构图模块2002基于分割结果快速识别出预览图像中是否包含海水。
示例性地，第一方法可以是基于边缘检测的方法、基于小波变换的方法、基于深度学习的方法等。考虑到性能要求高，精度要求较低，第一方法可以尽量选用传统分割方法或者模型较小的方法。例如，构图模块2002可以采用深度学习分割网络U-NET对预览图像进行分割。
另一种实施方式中，构图模块2002，具体用于对预览图像进行显著性检测，提取预览图像的显著性结果；将预览图像的显著性结果和场景信息输入训练好的人工智能AI网络，得到AI网络输出的预览图像对应的多种构图方式的概率分布；根据AI网络 输出的预览图像对应的多种构图方式的概率分布，确定与待拍摄场景匹配的构图方式对应的构图辅助线。
对某种类型的待拍摄场景而言，AI网络输出的概率分布中，不适合该类型的待拍摄场景的构图方式的概率为0或接近于0。
一些实施例中，构图模块2002，具体用于对预览图像进行矫正，得到矫正后的预览图像；根据矫正后的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线。
示例性地，云端包括线条检测网络和图像矫正模块(具体可以参见前述图2A中所示)；构图模块2002，具体用于将预览图像输入线条检测网络，通过线条检测网络检测预览图像中包含的线条；通过图像矫正模块根据预览图像中包含的线条，确定对预览图像进行矫正所需的变换矩阵，并采用变换矩阵对预览图像进行矫正，得到矫正后的预览图像。
可选地，线条检测网络包括：主干网络、连接点预测模块、线段采样模块、以及线段校正模块(具体可以参见前述图2B中所示)。构图模块2002，具体用于将预览图像输入主干网络，主干网络提取预览图像中的特征，输出预览图像对应的共享卷积特征图至连接点预测模块；连接点预测模块根据共享卷积特征图输出预览图像对应的候选连接点，并传输给线段采样模块；线段采样模块根据候选连接点预测出预览图像包含的线条。
可选地，变换矩阵至少包括旋转矩阵、单应性矩阵。
可选地，构图模块2002，还用于获取预览图像中包含的元素的轮廓线；根据构图辅助线、以及预览图像中包含的元素的轮廓线，生成适合待拍摄场景的拍摄轮廓线。发送模块2003，还用于向终端设备发送拍摄轮廓线。
可选地，图21示出了本申请实施例提供的电子设备的又一结构示意图。如图21所示，该电子设备还可以包括：构图优化模块2004。
接收模块2001，还用于接收来自终端设备的待拍摄场景对应的照片。构图优化模块2004，用于对所述照片进行构图优化，得到所述照片对应的至少一张构图优化后的照片。发送模块2003，还用于向终端设备发送所述至少一张构图优化后的照片。
示例性地，云端包括线条检测网络、图像矫正模块、显著性检测模块、以及美学评分网络(具体请参见前述图15中所示)。构图优化模块2004，具体用于将所述照片输入线条检测网络，通过线条检测网络检测所述照片中包含的线条；通过图像矫正模块根据所述照片中包含的线条，确定对所述照片进行矫正所需的变换矩阵，并采用变换矩阵对所述照片进行矫正，得到矫正后的照片；通过显著性检测模块对所述矫正后的照片进行显著性检测，得到所述矫正后的照片的显著性结果；将所述矫正后的照片、以及显著性结果输入美学评分网络，得到美学评分网络输出的多张候选裁切图、以及每张所述候选裁切图对应的评分；确定多张候选裁切图中评分最高的至少一张候选裁切图为构图优化后的照片。
一些实施例中，接收模块2001接收到的来自终端设备的初始照片为非人像照片，如：可以是一些风景/风光照片。
本申请实施例还提供一种终端设备，该终端设备可以包括拍照装置。该拍照装置可以用于实现前述实施例中所述的拍照方法中终端设备所执行的功能。该拍照装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元。
例如，图28示出了本申请实施例提供的拍照装置的结构示意图。如图28所示，该拍照装置可以包括：相机模块2801，发送模块2802，接收模块2803，显示模块2804，处理模块2805等。
其中，相机模块2801，用于获取待编辑照片和所述待编辑照片对应的场景信息。发送模块2802，用于向所述云端发送所述待编辑照片和场景信息，所述待编辑照片和场景信息用于所述云端对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片。接收模块2803，用于从云端接收至少一张构图优化后的照片。显示模块2804，用于显示所述待编辑照片、以及所述至少一张构图优化后的照片。处理模块2805，用于响应于用户对第一照片的选择操作，选择第一照片。第一照片包括所述待编辑照片、以及所述至少一张构图优化后的照片中的一张或多张。发送模块2802，还用于向云端发送第一照片，所述第一照片用于所述云端获取用户的美学偏好。
可选地，处理模块2805，还用于通过预设的卷积神经网络提取第一照片的图像特征。发送模块2802，还用于向所述云端发送所述第一照片的图像特征。
可选地，发送模块2802，还用于向所述云端发送所述第一照片的标识信息。
本申请实施例还提供一种云端服务器，该云端服务器可以用于实现前述实施例中所述的拍照方法中云端所执行的功能。该拍照装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元。
例如，图29示出了本申请实施例提供的电子设备的另一结构示意图。如图29所示，该拍照装置可以包括：接收模块2901，处理模块2902，发送模块2903等。
其中，接收模块2901，用于接收来自终端设备的待编辑照片和场景信息。处理模块2902，用于根据所述待编辑照片和场景信息，对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片。发送模块2903，用于将所述至少一张构图优化后的照片发送给所述终端设备。所述接收模块2901，还用于接收来自终端设备的第一照片，所述第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张。所处理模块2902，还用于根据所述第一照片，获取用户的美学偏好。
可选地，云端包括线条检测网络、图像矫正模块、显著性检测模块、以及美学评分网络；处理模块2902，还用于将所述待编辑图像输入所述线条检测网络，通过所述线条检测网络检测所述待编辑图像中包含的线条。处理模块2902，还用于通过所述图像矫正模块根据所述待编辑图像中包含的线条，确定对所述预览图像进行矫正所需的变换矩阵，并采用所述变换矩阵对所述待编辑图像进行矫正，得到矫正后的待编辑图像。
可选地，所述线条检测网络包括：主干网络、连接点预测模块、线段采样模块、以及线段校正模块。所述线条检测网络检测所述待编辑图像中包含的线条，包括：将所述待编辑图像输入主干网络后，所述主干网络提取所述待编辑图像中的特征，输出所述待编辑图像对应的共享卷积特征图至所述连接点预测模块；所述连接点预测模块根据所述共享卷积特征图输出所述待编辑图像对应的候选连接点，并传输给所述线段采样模块；所述线段采样模块根据所述候选连接点预测出所述待编辑图像包含的线条。
可选地，处理模块2902，还用于根据所述第一照片的标识信息获取存储在云端的对应所述标识信息的第一照片，并根据所述对应所述标识信息的照片获取用户的美学偏好。
本申请实施例还提供一种终端设备，该终端设备可以包括拍照装置。该拍照装置可以用于实现前述实施例中所述的拍照方法中终端设备所执行的功能。该拍照装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元。
例如，图30示出了本申请实施例提供的拍照装置的结构示意图。如图30所示，该拍照装置可以包括：相机模块3001，处理模块3002，发送模块3003，接收模块3004等。
其中，相机模块3001，用于获取待编辑照片。处理模块3002，用于响应于用户的编辑操作，对待编辑照片进行编辑，获取编辑后的照片。发送模块3003，用于向云端发送所述待编辑照片和编辑后的照片，所述待编辑照片和编辑后的照片用于所述云端获取用户的美学偏好，并根据用户的美学偏好获取基础画质优化参数。接收模块3004，用于从云端接收基础画质优化参数。处理模块3002，还用于根据所述基础画质优化参数更新本地的基础画质算法。
可选地，处理模块3002，还用于根据终端设备本地的基础画质算法处理所述待编辑照片，获取中间照片；并用于对所述中间照片进行编辑，获取编辑后的照片。
可选地，处理模块3002，还用于通过预设的卷积神经网络提取所述待编辑照片的图像特征和编辑后的照片的图像特征。发送模块3003，还用于向所述云端发送所述待编辑照片和编辑后的照片的图像特征。
本申请实施例还提供一种云端服务器，该云端服务器可以用于实现前述实施例中所述的拍照方法中云端服务器所执行的功能。该拍照装置的功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元。
例如，图31示出了本申请实施例提供的电子设备的另一结构示意图。如图31所示，该电子设备可以包括：接收模块3101，处理模块3102，发送模块3103等。
其中，接收模块3101，用于接收来自终端设备的待编辑照片和编辑后的照片。处理模块3102，用于根据所述待编辑照片和编辑后的照片，获取用户的美学偏好，并根据用户的美学偏好获取基础画质优化参数。发送模块3103，用于将所述基础画质优化参数发送给所述终端设备。
可选地，处理模块3102，还用于对基础画质算法进行建模，得到基础画质算法参 数与成像效果的关系函数。处理模块3102，还用于根据所述待编辑照片和编辑后的照片训练所述关系函数，获取用户的美学偏好。
应理解以上装置(云端的拍照装置或终端设备的拍照装置)中单元或模块(以下均称为单元)的划分仅仅是一种逻辑功能的划分，实际实现时可以全部或部分集成到一个物理实体上，也可以物理上分开。且装置中的单元可以全部以软件通过处理元件调用的形式实现；也可以全部以硬件的形式实现；还可以部分单元以软件通过处理元件调用的形式实现，部分单元以硬件的形式实现。
例如，各个单元可以为单独设立的处理元件，也可以集成在装置的某一个芯片中实现，此外，也可以以程序的形式存储于存储器中，由装置的某一个处理元件调用并执行该单元的功能。此外这些单元全部或部分可以集成在一起，也可以独立实现。这里所述的处理元件又可以称为处理器，可以是一种具有信号的处理能力的集成电路。在实现过程中，上述方法的各步骤或以上各个单元可以通过处理器元件中的硬件的集成逻辑电路实现或者以软件通过处理元件调用的形式实现。
在一个例子中，以上装置中的单元可以是被配置成实施以上方法的一个或多个集成电路，例如：一个或多个专用集成电路(application specific integrated circuit，ASIC)，或，一个或多个数字信号处理器(digital signal process，DSP)，或，一个或者多个现场可编辑逻辑门阵列(field programmable gate array，FPGA)，或这些集成电路形式中至少两种的组合。
再如，当装置中的单元可以通过处理元件调度程序的形式实现时，该处理元件可以是通用处理器，例如CPU或其它可以调用程序的处理器。再如，这些单元可以集成在一起，以片上系统(system-on-a-chip，SOC)的形式实现。
在一种实现中，以上装置实现以上方法中各个对应步骤的单元可以通过处理元件调度程序的形式实现。例如，该装置可以包括处理元件和存储元件，处理元件调用存储元件存储的程序，以执行以上方法实施例所述的方法。存储元件可以为与处理元件处于同一芯片上的存储元件，即片内存储元件。
在另一种实现中，用于执行以上方法的程序可以在与处理元件处于不同芯片上的存储元件，即片外存储元件。此时，处理元件从片外存储元件调用或加载程序于片内存储元件上，以调用并执行以上方法实施例所述的方法。
例如，本申请实施例还可以提供一种装置，如：电子设备，可以包括：处理器，用于存储该处理器可执行指令的存储器。该处理器被配置为执行上述指令时，使得该电子设备实现如前述实施例所述的拍照方法中终端设备执行的步骤。该存储器可以位于该电子设备之内，也可以位于该电子设备之外。且该处理器包括一个或多个。
该电子设备可以是手机、平板电脑、可穿戴设备、车载设备、AR/VR设备、笔记本电脑、UMPC、上网本、PDA等移动终端，或者，也可以是数码相机、单反相机/微单相机、运动摄像机、云台相机、无人机等专业的拍摄设备。
又例如，本申请实施例还可以提供一种装置，如：电子设备，可以包括：处理器，用于存储该处理器可执行指令的存储器。该处理器被配置为执行上述指令时，使得该电子设备实现如前述实施例所述的拍照方法中云端执行的步骤。该存储器可以位于该 电子设备之内，也可以位于该电子设备之外。且该处理器包括一个或多个。
该电子设备可以是计算机、服务器、或者多个服务器组成的服务器集群等。
在又一种实现中，该装置实现以上方法中各个步骤的单元可以是被配置成一个或多个处理元件，这里的处理元件可以为集成电路，例如：一个或多个ASIC，或，一个或多个DSP，或，一个或者多个FPGA，或者这些类集成电路的组合。这些集成电路可以集成在一起，构成芯片。
例如，本申请实施例还提供一种芯片，该芯片可以应用于电子设备。芯片包括一个或多个接口电路和一个或多个处理器；接口电路和处理器通过线路互联；处理器通过接口电路从电子设备的存储器接收并执行计算机指令，以实现如前述实施例所述的拍照方法中终端设备执行的步骤。
又例如，本申请实施例还提供一种芯片，该芯片可以应用于电子设备。芯片包括一个或多个接口电路和一个或多个处理器；接口电路和处理器通过线路互联；处理器通过接口电路从电子设备的存储器接收并执行计算机指令，以实现如前述实施例所述的拍照方法中云端执行的步骤。
可选地，本申请实施例还提供一种计算机程序产品，包括计算机可读代码，当计算机可读代码在电子设备中运行时，使得电子设备实现如前述实施例所述的拍照方法中终端设备执行的步骤。
可选地，本申请实施例还提供一种计算机程序产品，包括计算机可读代码，当计算机可读代码在电子设备中运行时，使得电子设备实现如前述实施例所述的拍照方法中云端执行的步骤。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时， 可以存储在一个可读取存储介质中。
基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，如：程序。该软件产品存储在一个程序产品，如计算机可读存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
例如，本申请实施例还可以提供一种计算机可读存储介质，其上存储有计算机程序指令。当计算机程序指令被电子设备执行时，使得电子设备实现如前述实施例所述的拍照方法中终端设备执行的步骤。
又例如，本申请实施例还可以提供一种计算机可读存储介质，其上存储有计算机程序指令。当计算机程序指令被电子设备执行时，使得电子设备实现如前述实施例所述的拍照方法中云端执行的步骤。
可选地，本申请还有一些实施例中，前述实施例中所述的云端执行的所有功能，也可以全部集成在终端设备中完成。
例如，本申请实施例还可以提供一种拍照方法，该拍照方法可以应用于终端设备。该方法可以包括：终端设备启动运行拍照应用程序后，获取待拍摄场景对应的预览图像，并在拍摄界面进行显示；同时，终端设备通过传感器采集待拍摄场景的场景信息。终端设备根据预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线。终端设备将构图辅助线与正在显示的预览图像一同显示在拍摄界面。终端设备接收到用户的拍照操作后，响应于用户的拍照操作，获取待拍摄场景对应的照片。
该拍照方法也能够在用户使用终端设备进行拍照时，通过构图辅助线对用户进行构图引导。用户在使用终端设备进行拍照时，能够根据云端推荐的构图辅助线，更轻松地知道通过哪种构图方式进行构图，不需要自己判断构图方式，也不要复杂的构图操作。如：构图辅助线可以为用户指示适合待拍摄场景的构图方式，引导用户按照构图辅助线指示的构图方式对终端设备的拍摄位置、角度等进行调整以完成构图，从而可以使得没有丰富拍摄经验的用户在进行拍照时也能够完成较好的构图，用户的体验可以更好。
类似地，终端设备根据预览图像和场景信息生成适合待拍摄场景的构图方式对应的构图辅助线之前，也可以先对预览图像进行矫正，得到矫正后的预览图像。然后，终端设备具体可以根据矫正后的预览图像和场景信息，生成适合待拍摄场景的构图方式对应的构图辅助线。
可选地，终端设备还可以获取预览图像中包含的元素的轮廓线；并根据构图辅助线、以及预览图像中包含的元素的轮廓线，生成适合待拍摄场景的拍摄轮廓线。然后，终端设备可以将拍摄轮廓线、构图辅助线与正在显示的预览图像一同显示在拍摄界面。
可选地，终端设备接收到用户的拍照操作后，响应于用户的拍照操作，获取待拍摄场景对应的照片之后，还可以在本地对照片进行构图优化，并显示所述照片、以及所述构图优化后的照片。
需要说明的是，当前述实施例中所述的云端执行的所有功能，被全部集成在终端设备中完成时，这些功能在终端设备中的具体实现原理与在云端中实现的原理可以相同。
例如，终端设备根据预览图像和场景信息生成适合待拍摄场景的构图方式对应的构图辅助线的基本原理，与前述实施例中云端根据预览图像和场景信息生成适合待拍摄场景的构图方式对应的构图辅助线的基本原理相同。终端设备生成拍摄轮廓线的原理与云端生成拍摄轮廓线的原理相同。终端设备进行构图优化的原理与云端进行构图优化的原理相同等。在此不再一一赘述。
另外，当前述实施例中所述的云端执行的所有功能，被全部集成在终端设备中完成时，也可以达到与前述实施例相同或相近的技术效果。其区别仅仅在于，本实施例中，终端设备的计算能力需要支持实现云端执行的所有功能。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (49)

  1. 一种拍照方法，其特征在于，所述方法应用于端云协同系统，所述端云协同系统包括终端设备和云端；所述终端设备通过无线网络与所述云端连接；所述方法包括：
    所述终端设备响应于用户的操作，获取并显示待拍摄场景对应的预览图像；
    所述终端设备采集所述待拍摄场景的场景信息；
    所述终端设备向所述云端发送获取到的预览图像和场景信息；
    所述云端根据所述预览图像和所述场景信息，生成适合所述待拍摄场景的构图方式对应的构图辅助线，并将所述构图辅助线发送给所述终端设备；
    所述终端设备接收到所述构图辅助线后，将所述构图辅助线与所述预览图像显示在所述拍摄界面；
    所述终端设备响应于用户的拍照操作，获取所述待拍摄场景对应的照片。
  2. 根据权利要求1所述的方法，其特征在于，所述云端根据所述预览图像和所述场景信息，生成适合所述待拍摄场景的构图方式对应的构图辅助线，包括：
    所述云端根据所述预览图像和所述场景信息，识别所述预览图像中包含的元素，并记录所述预览图像中不同元素的位置和所占比例；
    所述云端根据所述预览图像中包含的元素，以及不同元素的位置和所占比例，按照预设的匹配规则确定与所述待拍摄场景匹配的构图方式对应的构图辅助线；
    其中，所述匹配规则包括至少一种类型的待拍摄场景与构图辅助线之间的对应关系，不同类型的待拍摄场景中包含的元素、以及不同元素的位置和所占比例不同。
  3. 根据权利要求2所述的方法，其特征在于，所述云端根据所述预览图像和所述场景信息，识别所述预览图像中包含的元素，包括：
    所述云端采用第一方法对所述预览图像进行分割，然后基于分割结果、结合所述场景信息识别所述预览图像中包含的元素；
    其中，所述场景信息用于辅助所述云端基于所述分割结果快速识别所述预览图像中包含的元素。
  4. 根据权利要求1所述的方法，其特征在于，所述云端根据所述预览图像和所述场景信息，生成适合所述待拍摄场景的构图方式对应的构图辅助线，包括：
    所述云端对所述预览图像进行显著性检测，提取所述预览图像的显著性结果；
    所述云端将所述预览图像的显著性结果和所述场景信息输入训练好的人工智能AI网络，得到所述AI网络输出的所述预览图像对应的多种构图方式的概率分布；
    所述云端根据所述AI网络输出的所述预览图像对应的多种构图方式的概率分布，确定与所述待拍摄场景匹配的构图方式对应的构图辅助线。
  5. 根据权利要求1-4任一项所述的方法，其特征在于，所述方法还包括：
    所述终端设备向所述云端发送所述待拍摄场景对应的照片；
    所述云端对所述照片进行构图优化，得到所述照片对应的至少一张构图优化后的照片，并将所述至少一张构图优化后的照片发送给所述终端设备；
    所述终端设备显示所述照片、以及所述至少一张构图优化后的照片；
    所述终端设备响应于用户的保存操作，保存第一照片，所述第一照片包括所述照 片、以及所述至少一张构图优化后的照片中的一张或多张。
  6. 根据权利要求5所述的方法，其特征在于，所述云端包括线条检测网络、图像矫正模块、显著性检测模块、以及美学评分网络；所述云端对所述照片进行构图优化，得到所述照片对应的至少一张构图优化后的照片，包括：
    所述云端将所述照片输入所述线条检测网络，通过所述线条检测网络检测所述照片中包含的线条；
    所述云端通过所述图像矫正模块根据所述照片中包含的线条，确定对所述照片进行矫正所需的变换矩阵，并采用所述变换矩阵对所述照片进行矫正，得到矫正后的照片；
    所述云端通过所述显著性检测模块对所述矫正后的照片进行显著性检测，得到所述矫正后的照片的显著性结果；
    所述云端将所述矫正后的照片、以及所述显著性结果输入所述美学评分网络，得到所述美学评分网络输出的多张候选裁切图、以及每张所述候选裁切图对应的评分；
    所述云端确定多张所述候选裁切图中评分最高的至少一张候选裁切图为所述构图优化后的照片。
  7. 根据权利要求5或6所述的方法，其特征在于，所述方法还包括：
    所述终端设备向所述云端发送所述第一照片；
    所述云端根据所述第一照片获取用户的美学偏好。
  8. 根据权利要求7所述的方法，其特征在于，所述终端设备向所述云端发送所述第一照片，包括：
    所述终端设备通过预设的卷积神经网络提取所述第一照片的图像特征；
    所述终端设备向所述云端发送所述第一照片的图像特征。
  9. 根据权利要求7所述的方法，其特征在于，
    所述终端设备向所述云端发送所述第一照片，包括：所述终端设备向所述云端发送所述第一照片的标识信息；
    所述云端根据所述第一照片获取用户的美学偏好，包括：所述云端根据所述第一照片的标识信息获取存储在云端的第一照片，并根据所述第一照片获取用户的美学偏好。
  10. 根据权利要求7-9任一项所述的方法，其特征在于，所述方法还包括：
    所述云端根据所述第一照片获取用户的美学偏好，包括：
    所述云端根据所述第一照片获取对应的场景信息，并根据所述第一照片和场景信息获取用户的美学偏好。
  11. 一种拍照方法，其特征在于，所述方法应用于终端设备，所述终端设备通过无线网络与云端连接；所述方法包括：
    所述终端设备响应于用户的操作，获取并显示待拍摄场景对应的预览图像；
    所述终端设备采集所述待拍摄场景的场景信息；
    所述终端设备向所述云端发送获取到的预览图像和场景信息；
    所述终端设备接收来自所述云端的构图辅助线，所述构图辅助线用于指示适合所 述待拍摄场景的构图方式；
    所述终端设备将所述构图辅助线与正在显示的预览图像显示在所述拍摄界面；
    所述终端设备响应于用户的拍照操作，获取所述待拍摄场景对应的照片。
  12. 根据权利要求10所述的方法，其特征在于，所述终端设备向所述云端发送场景信息的频率，小于所述终端设备向所述云端发送预览图像的频率。
  13. 根据权利要求11或12所述的方法，其特征在于，所述终端设备通过传感器采集所述待拍摄场景的场景信息，所述传感器至少包括位置传感器、气压传感器，温度传感器，环境光传感器；
    所述场景信息至少包括所述待拍摄场景对应的位置信息、气压信息、温度信息、光强信息。
  14. 根据权利要求11-13任一项所述的方法，其特征在于，所述方法还包括：
    所述终端设备响应于用户关闭所述构图辅助线的显示功能的操作，不显示所述构图辅助线。
  15. 根据权利要求11-14任一项所述的方法，其特征在于，所述终端设备向所述云端发送获取到的预览图像，包括：
    所述终端设备通过预设的卷积神经网络提取获取到的预览图像的图像特征；
    所述终端设备向所述云端发送获取到的预览图像的图像特征。
  16. 根据权利要求11-15任一项所述的方法，其特征在于，所述方法还包括：
    所述终端设备接收来自所述云端的拍摄轮廓线；
    所述终端设备将所述拍摄轮廓线、所述构图辅助线与所述预览图像显示在所述拍摄界面。
  17. 根据权利要求11-15任一项所述的方法，其特征在于，所述方法还包括：
    所述终端设备向所述云端发送所述待拍摄场景对应的照片；
    所述终端设备接收来自所述云端的所述照片对应的至少一张构图优化后的照片；
    所述终端设备显示所述照片、以及所述至少一张构图优化后的照片；
    所述终端设备响应于用户对第一照片的保存操作，保存所述第一照片，所述第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张。
  18. 根据权利要求17所述的方法，其特征在于，所述方法还包括：
    所述终端设备向所述云端发送所述第一照片，所述第一照片用于所述云端获取用户的美学偏好。
  19. 根据权利要求18所述的方法，其特征在于，所述终端设备向所述云端发送所述第一照片，包括：
    所述终端设备通过预设的卷积神经网络提取所述第一照片的图像特征；
    所述终端设备向所述云端发送所述第一照片的图像特征。
  20. 一种拍照方法，其特征在于，所述方法应用于云端，所述云端通过无线网络与终端设备连接；所述方法包括：
    所述云端接收来自所述终端设备的待拍摄场景对应的预览图像和场景信息；
    所述云端根据所述预览图像和所述场景信息，生成适合所述待拍摄场景的构图方 式对应的构图辅助线；
    所述云端向所述终端设备发送所述构图辅助线。
  21. 根据权利要求20所述的方法，其特征在于，所述云端根据所述预览图像和所述场景信息，生成适合所述待拍摄场景的构图方式对应的构图辅助线之前，所述方法还包括：
    所述云端对所述预览图像进行矫正，得到矫正后的预览图像；
    所述云端根据所述预览图像和所述场景信息，生成适合所述待拍摄场景的构图方式对应的构图辅助线，包括：
    所述云端根据所述矫正后的预览图像和所述场景信息，生成适合所述待拍摄场景的构图方式对应的构图辅助线。
  22. 根据权利要求21所述的方法，其特征在于，所述云端包括线条检测网络和图像矫正模块；所述云端对所述预览图像进行矫正，得到矫正后的预览图像，包括：
    所述云端将所述预览图像输入所述线条检测网络，通过所述线条检测网络检测所述预览图像中包含的线条；
    所述云端通过所述图像矫正模块根据所述预览图像中包含的线条，确定对所述预览图像进行矫正所需的变换矩阵，并采用所述变换矩阵对所述预览图像进行矫正，得到矫正后的预览图像。
  23. 根据权利要求22所述的方法，其特征在于，所述线条检测网络包括：主干网络、连接点预测模块、线段采样模块、以及线段校正模块；
    所述线条检测网络检测所述预览图像中包含的线条，包括：
    将所述预览图像输入主干网络后，所述主干网络提取所述预览图像中的特征，输出所述预览图像对应的共享卷积特征图至所述连接点预测模块；
    所述连接点预测模块根据所述共享卷积特征图输出所述预览图像对应的候选连接点，并传输给所述线段采样模块；
    所述线段采样模块根据所述候选连接点预测出所述预览图像包含的线条。
  24. 根据权利要求22或23所述的方法，其特征在于，所述变换矩阵至少包括旋转矩阵、单应性矩阵。
  25. 根据权利要求20-24任一项所述的方法，其特征在于，所述云端根据所述预览图像和所述场景信息，生成适合所述待拍摄场景的构图方式对应的构图辅助线之后，所述方法还包括：
    所述云端获取所述预览图像中包含的元素的轮廓线；
    所述云端根据所述构图辅助线、以及所述预览图像中包含的元素的轮廓线，生成适合所述待拍摄场景的拍摄轮廓线；
    所述云端向所述终端设备发送所述拍摄轮廓线。
  26. 根据权利要求20-25任一项所述的方法，其特征在于，所述方法还包括：
    所述云端接收来自所述终端设备的所述待拍摄场景对应的照片；
    所述云端对所述照片进行构图优化，得到所述照片对应的至少一张构图优化后的照片；
    所述云端向所述终端设备发送所述至少一张构图优化后的照片。
  27. 根据权利要求26所述的方法，其特征在于，所述方法还包括：
    所述云端接收终端设备发送的第一照片，所述第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张；
    所述云端根据所述第一照片，获取用户的美学偏好。
  28. 一种拍照方法，其特征在于，所述方法应用于端云协同系统，所述端云协同系统包括终端设备和云端；所述终端设备通过无线网络与所述云端连接；所述方法包括：
    所述终端设备获取待编辑照片和待编辑照片对应的场景信息；
    所述终端设备向所述云端发送所述待编辑照片和场景信息；
    所述云端根据所述待编辑照片和场景信息，对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片，并将所述至少一张构图优化后的照片发送给所述终端设备；
    所述终端设备显示所述待编辑照片、以及所述至少一张构图优化后的照片；
    所述终端设备响应于用户的操作，选择第一照片，所述第一照片包括所述待编辑照片、以及所述至少一张构图优化后的照片中的一张或多张；
    所述终端设备向所述云端发送所述第一照片；
    所述云端根据所述第一照片获取用户的美学偏好。
  29. 根据权利要求28所述的方法，其特征在于，所述云端包括线条检测网络、图像矫正模块、显著性检测模块、以及美学评分网络；所述云端对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片，包括：
    所述云端将所述待编辑图像输入所述线条检测网络，通过所述线条检测网络检测所述待编辑图像中包含的线条；
    所述云端通过所述图像矫正模块根据所述待编辑图像中包含的线条，确定对所述预览图像进行矫正所需的变换矩阵，并采用所述变换矩阵对所述待编辑图像进行矫正，得到矫正后的待编辑图像。
  30. 根据权利要求29所述的方法，其特征在于，所述线条检测网络包括：主干网络、连接点预测模块、线段采样模块、以及线段校正模块；
    所述线条检测网络检测所述待编辑图像中包含的线条，包括：
    将所述待编辑图像输入主干网络后，所述主干网络提取所述待编辑图像中的特征，输出所述待编辑图像对应的共享卷积特征图至所述连接点预测模块；
    所述连接点预测模块根据所述共享卷积特征图输出所述待编辑图像对应的候选连接点，并传输给所述线段采样模块；
    所述线段采样模块根据所述候选连接点预测出所述待编辑图像包含的线条。
  31. 一种拍照方法，其特征在于，所述方法应用于终端设备，所述终端设备通过无线网络与云端连接；所述方法包括：
    所述终端设备响应于用户的操作，获取待编辑照片和所述待编辑照片对应的场景信息；
    所述终端设备向所述云端发送所述待编辑照片和场景信息，所述待编辑照片和场景信息用于所述云端对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片；
    所述终端设备接收所述云端发送的所述至少一张构图优化后的照片；
    所述终端设备显示所述待编辑照片、以及所述至少一张构图优化后的照片；
    所述终端设备响应于用户的选择操作，选择第一照片，所述第一照片包括所述待编辑照片、以及所述至少一张构图优化后的照片中的一张或多张；
    所述终端设备向所述云端发送所述第一照片，所述第一照片用于所述云端获取用户的美学偏好。
  32. 根据权利要求31所述的方法，其特征在于，所述终端设备向所述云端发送所述第一照片，包括：
    所述终端设备通过预设的卷积神经网络提取所述第一照片的图像特征；
    所述终端设备向所述云端发送所述第一照片的图像特征。
  33. 根据权利要求31或32所述的方法，其特征在于，
    所述终端设备向所述云端发送所述第一照片，包括：所述终端设备向所述云端发送所述第一照片的标识信息。
  34. 根据权利要求31-33任一项所述的方法，其特征在于，
    所述场景信息是所述待编辑照片在拍摄时通过传感器采集的，所述传感器至少包括位置传感器、气压传感器，温度传感器，环境光传感器；
    所述场景信息至少包括所述待拍摄场景对应的位置信息、气压信息、温度信息、光强信息。
  35. 一种拍照方法，所述方法应用于云端，所述云端通过无线网络与终端设备连接；所述方法包括：
    所述云端接收来自所述终端设备的待编辑照片和场景信息；
    所述云端根据所述待编辑照片和场景信息，对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片，并将所述至少一张构图优化后的照片发送给所述终端设备；
    所述云端接收来自所述终端设备的第一照片，所述第一照片包括所述照片、以及所述至少一张构图优化后的照片中的一张或多张；
    所述云端根据所述第一照片，获取用户的美学偏好。
  36. 根据权利要求35所述的方法，其特征在于，所述云端包括线条检测网络、图像矫正模块、显著性检测模块、以及美学评分网络；所述云端对所述待编辑照片进行构图优化，得到所述待编辑照片对应的至少一张构图优化后的照片，包括：
    所述云端将所述待编辑图像输入所述线条检测网络，通过所述线条检测网络检测所述待编辑图像中包含的线条；
    所述云端通过所述图像矫正模块根据所述待编辑图像中包含的线条，确定对所述预览图像进行矫正所需的变换矩阵，并采用所述变换矩阵对所述待编辑图像进行矫正，得到矫正后的待编辑图像。
  37. 根据权利要求36所述的方法，其特征在于，所述线条检测网络包括：主干网络、连接点预测模块、线段采样模块、以及线段校正模块；
    所述线条检测网络检测所述待编辑图像中包含的线条，包括：
    将所述待编辑图像输入主干网络后，所述主干网络提取所述待编辑图像中的特征，输出所述待编辑图像对应的共享卷积特征图至所述连接点预测模块；
    所述连接点预测模块根据所述共享卷积特征图输出所述待编辑图像对应的候选连接点，并传输给所述线段采样模块；
    所述线段采样模块根据所述候选连接点预测出所述待编辑图像包含的线条。
  38. 根据权利要求35-37任一项所述的方法，其特征在于，所述云端根据所述第一照片获取用户的美学偏好，包括：
    所述云端根据所述第一照片的标识信息获取存储在云端的对应所述标识信息的照片，并根据所述对应所述标识信息的照片获取用户的美学偏好。
  39. 一种拍照方法，其特征在于，所述方法应用于端云协同系统，所述端云协同系统包括终端设备和云端；所述终端设备通过无线网络与所述云端连接；所述方法包括：
    所述终端设备获取待编辑照片；
    所述终端设备响应于用户的编辑操作，对待编辑照片进行编辑，获取编辑后的照片；
    所述终端设备向所述云端发送所述待编辑照片和编辑后的照片；
    所述云端根据所述待编辑照片和编辑后的照片，获取用户的美学偏好，并根据用户的美学偏好获取基础画质优化参数；
    所述云端将所述基础画质优化参数发送给所述终端设备；
    所述终端设备根据所述基础画质优化参数更新本地的基础画质算法。
  40. 根据权利要求39所述的方法，其特征在于，在所述终端设备响应于用户的编辑操作，对待编辑照片进行编辑，获取编辑后的照片，包括：
    所述终端设备根据本地的基础画质算法处理所述待编辑照片，获取中间照片；
    所述终端对所述中间照片进行编辑，获取编辑后的照片。
  41. 根据权利要求39或40所述的方法，其特征在于，所述终端设备向所述云端发送所述待编辑照片和编辑后的照片，包括：
    所述终端设备通过预设的卷积神经网络提取所述待编辑照片的图像特征和编辑后的照片的图像特征；
    所述终端设备向所述云端发送所述待编辑照片和编辑后的照片的图像特征。
  42. 一种拍照方法，其特征在于，所述方法应用于终端设备，所述终端设备通过无线网络与云端连接；所述方法包括：
    所述终端设备获取待编辑照片；
    所述终端设备响应于用户的编辑操作，对待编辑照片进行编辑，获取编辑后的照片；
    所述终端设备向所述云端发送所述待编辑照片和编辑后的照片，所述待编辑照片 和编辑后的照片用于所述云端获取用户的美学偏好，并根据用户的美学偏好获取基础画质优化参数；
    所述终端设备接收来自所述云端的所述基础画质优化参数；
    所述终端设备根据所述基础画质优化参数更新本地的基础画质算法。
  43. 根据权利要求42所述的方法，其特征在于，在所述终端设备响应于用户的编辑操作，对待编辑照片进行编辑，获取编辑后的照片，包括：
    所述终端设备根据本地的基础画质算法处理所述待编辑照片，获取中间照片；
    所述终端对所述中间照片进行编辑，获取编辑后的照片。
  44. 根据权利要求42或43所述的方法，其特征在于，所述终端设备向所述云端发送所述待编辑照片和编辑后的照片，包括：
    所述终端设备通过预设的卷积神经网络提取所述待编辑照片的图像特征和编辑后的照片的图像特征；
    所述终端设备向所述云端发送所述待编辑照片和编辑后的照片的图像特征。
  45. 根据权利要求42-44任一项所述的方法，其特征在于，所述基础画质算法包括美颜算法、滤镜算法、曝光算法或者降噪算法。
  46. 一种拍照方法，所述方法应用于云端，所述云端通过无线网络与终端设备连接；所述方法包括：
    所述云端接收来自终端设备的待编辑照片和编辑后的照片；
    所述云端根据所述待编辑照片和编辑后的照片，获取用户的美学偏好，并根据用户的美学偏好获取基础画质优化参数；
    所述云端将所述基础画质优化参数发送给所述终端设备。
  47. 根据权利要求46所述的方法，其特征在于，所述云端根据所述待编辑照片和编辑后的照片，获取用户的美学偏好，包括：
    所述云端对基础画质算法进行建模，得到基础画质算法参数与成像效果的关系函数；
    所述云端根据所述待编辑照片和编辑后的照片训练所述关系函数，获取用户的美学偏好。
  48. 一种电子设备，其特征在于，包括：处理器，用于存储所述处理器可执行指令的存储器；
    所述处理器被配置为执行所述指令时，使得所述电子设备实现如权利要求11-19任一项所述的方法；或者，如权利要求20-27任一项所述的方法；或者，如权利要求31-34任一项所述的方法；或者，如权利要求35-38任一项所述的方法；或者，如权利要求42-45任一项所述的方法；或者，如权利要求46或47所述的方法。
  49. 一种计算机可读存储介质，其上存储有计算机程序指令；其特征在于，
    当所述计算机程序指令被电子设备执行时，使得电子设备实现如权利要求11-19任一项所述的方法，或者，如权利要求20-27任一项所述的方法；或者，如权利要求31-34任一项所述的方法；或者，如权利要求35-38任一项所述的方法；或者，如权利要求42-45任一项所述的方法；或者，如权利要求46或47所述的方法。
PCT/CN2022/074348 2021-04-26 2022-01-27 拍照方法及装置 WO2022227752A1 (zh)

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
CN202110454641.4 2021-04-26
CN202110454641 2021-04-26
CN202110524967.X 2021-05-14
CN202110524967 2021-05-14
CN202110524940 2021-05-14
CN202110524940.0 2021-05-14
CN202110897914.2 2021-08-05
CN202110897914.2A CN114466133B (zh) 2021-04-26 2021-08-05 拍照方法及装置

Publications (1)

Publication Number Publication Date
WO2022227752A1 true WO2022227752A1 (zh) 2022-11-03

Family

ID=81406301

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/074348 WO2022227752A1 (zh) 2021-04-26 2022-01-27 拍照方法及装置

Country Status (2)

Country Link
CN (1) CN114466133B (zh)
WO (1) WO2022227752A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116546310A (zh) * 2023-07-05 2023-08-04 北京电子科技学院 基于人工智能的摄影辅助方法、装置、设备和介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116543004B (zh) * 2023-07-05 2024-04-19 荣耀终端有限公司 图像裁切方法、装置、终端设备及计算机可读存储介质

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103533244A (zh) * 2013-10-21 2014-01-22 深圳市中兴移动通信有限公司 拍摄装置及其自动视效处理拍摄方法
CN103945129A (zh) * 2014-04-30 2014-07-23 深圳市中兴移动通信有限公司 基于移动终端的拍照预览构图指导方法及系统
JP2015103968A (ja) * 2013-11-25 2015-06-04 オリンパス株式会社 画像処理装置、画像処理方法及び画像処理プログラム
CN106101536A (zh) * 2016-06-22 2016-11-09 维沃移动通信有限公司 一种拍照方法及移动终端
CN107566529A (zh) * 2017-10-18 2018-01-09 维沃移动通信有限公司 一种拍照方法、移动终端及云端服务器
CN111294502A (zh) * 2018-12-07 2020-06-16 中国移动通信集团终端有限公司 拍照方法、具有拍照功能的装置、设备及存储介质
CN111614897A (zh) * 2020-05-13 2020-09-01 南京邮电大学 一种基于用户偏好多维度驱动的智能拍照方法
CN112017193A (zh) * 2020-08-24 2020-12-01 杭州趣维科技有限公司 一种基于视觉显著性和美学分数的图像裁切装置及方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005321832A (ja) * 2005-07-20 2005-11-17 Olympus Corp カメラの画像表示方法
KR101643602B1 (ko) * 2009-06-24 2016-07-29 삼성전자주식회사 구도 가이드 방법 및 장치, 및 디지털 촬영 장치
JP2013162364A (ja) * 2012-02-06 2013-08-19 Canon Inc 撮像装置、その制御方法及びプログラム並びに記録媒体
CN103338329B (zh) * 2013-05-31 2017-04-19 广东欧珀移动通信有限公司 一种拍摄照片的方法及装置
CN104301613B (zh) * 2014-10-16 2016-03-02 深圳市中兴移动通信有限公司 移动终端及其拍摄方法
CN107592451A (zh) * 2017-08-31 2018-01-16 努比亚技术有限公司 一种多模式辅助拍照方法、装置及计算机可读存储介质
CN107835364A (zh) * 2017-10-30 2018-03-23 维沃移动通信有限公司 一种拍照辅助方法及移动终端
KR102438201B1 (ko) * 2017-12-01 2022-08-30 삼성전자주식회사 사진 촬영과 관련된 추천 정보를 제공하는 방법 및 시스템
CN108462826A (zh) * 2018-01-23 2018-08-28 维沃移动通信有限公司 一种辅助拍照的方法及移动终端

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103533244A (zh) * 2013-10-21 2014-01-22 深圳市中兴移动通信有限公司 拍摄装置及其自动视效处理拍摄方法
JP2015103968A (ja) * 2013-11-25 2015-06-04 オリンパス株式会社 画像処理装置、画像処理方法及び画像処理プログラム
CN103945129A (zh) * 2014-04-30 2014-07-23 深圳市中兴移动通信有限公司 基于移动终端的拍照预览构图指导方法及系统
CN106101536A (zh) * 2016-06-22 2016-11-09 维沃移动通信有限公司 一种拍照方法及移动终端
CN107566529A (zh) * 2017-10-18 2018-01-09 维沃移动通信有限公司 一种拍照方法、移动终端及云端服务器
CN111294502A (zh) * 2018-12-07 2020-06-16 中国移动通信集团终端有限公司 拍照方法、具有拍照功能的装置、设备及存储介质
CN111614897A (zh) * 2020-05-13 2020-09-01 南京邮电大学 一种基于用户偏好多维度驱动的智能拍照方法
CN112017193A (zh) * 2020-08-24 2020-12-01 杭州趣维科技有限公司 一种基于视觉显著性和美学分数的图像裁切装置及方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116546310A (zh) * 2023-07-05 2023-08-04 北京电子科技学院 基于人工智能的摄影辅助方法、装置、设备和介质
CN116546310B (zh) * 2023-07-05 2023-09-15 北京电子科技学院 基于人工智能的摄影辅助方法、装置、设备和介质

Also Published As

Publication number Publication date
CN114466133A (zh) 2022-05-10
CN114466133B (zh) 2023-02-10

Similar Documents

Publication Publication Date Title
US11870951B2 (en) Photographing method and terminal
WO2022227752A1 (zh) 拍照方法及装置
WO2021223500A1 (zh) 一种拍摄方法及设备
WO2019105297A1 (zh) 图像虚化处理方法、装置、移动设备及存储介质
WO2022001806A1 (zh) 图像变换方法和装置
US11388334B2 (en) Automatic camera guidance and settings adjustment
WO2019105298A1 (zh) 图像虚化处理方法、装置、移动设备及存储介质
WO2021180046A1 (zh) 图像留色方法及设备
WO2021185374A1 (zh) 一种拍摄图像的方法及电子设备
US20230353864A1 (en) Photographing method and apparatus for intelligent framing recommendation
CN113850709A (zh) 图像变换方法和装置
WO2023165535A1 (zh) 图像处理方法、装置及设备
CN114693538A (zh) 一种图像处理方法及装置
CN117177052B (zh) 图像获取方法、电子设备及计算机可读存储介质
CN114979458B (zh) 一种图像的拍摄方法及电子设备
CN114615421B (zh) 图像处理方法及电子设备
CN116437198B (zh) 图像处理方法与电子设备
EP4231621A1 (en) Image processing method and electronic device
CN117177052A (zh) 图像获取方法、电子设备及计算机可读存储介质
CN115623319A (zh) 一种拍摄方法及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22794233

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22794233

Country of ref document: EP

Kind code of ref document: A1