WO2021029648A1

WO2021029648A1 - Image capturing apparatus and auxiliary photographing method therefor

Info

Publication number: WO2021029648A1
Application number: PCT/KR2020/010610
Authority: WO
Inventors: Jun Li
Original assignee: Samsung Electronics Co., Ltd.
Priority date: 2019-08-14
Filing date: 2020-08-11
Publication date: 2021-02-18
Also published as: CN110493517A

Abstract

An auxiliary photographing method for an image capturing apparatus and an image capturing apparatus are provided. The auxiliary photographing method includes: acquiring scene information on a user's current photographing; acquiring an associated picture associated with the scene information from historical pictures photographed by the user based on the scene information; and providing an operation instruction based on an acquisition result of the associated picture.

Description

IMAGE CAPTURING APPARATUS AND AUXILIARY PHOTOGRAPHING METHOD THEREFOR

The present disclosure relates to an auxiliary photographing method for an image capturing apparatus and relates to an image capturing apparatus, more specifically to such an image capturing apparatus and auxiliary photographing method for the same that provide a photographing prompt based on a historically photographed picture.

With the development of technologies, users’ demands on image capturing apparatuses are gradually increasing. When the user uses an image capturing apparatus, he/she desires to photograph a souvenir picture. For example, the user expects to photograph a souvenir picture at the same location, in the same background (e.g., a building or landscape) and/or in the same pose every other year or every other time to show changes in his/her family members, which is very memorable for the user. However, in absence of assistance, due to the location and angle of the image capturing apparatus as well as the position and pose of the photographed object, etc., it would be easy to cause the newly photographed souvenir picture to have a poor background coincidence and/or pose coincidence compared with the original souvenir picture.

Therefore, a photographing prompt may be provided based on a picture historically photographed by the image capturing apparatus, so that the newly photographed picture and the historically photographed picture have higher background coincidence and/or pose coincidence compared with the historically photographed picture.

By the above auxiliary photographing method and the image capturing apparatus, it is possible to photograph a picture that is highly coincident with a historically photographed picture.

The present disclosure is proposed to at least address the above disadvantages and provide the advantages as described below.

According to an aspect of the present disclosure, an auxiliary photographing method for an image capturing apparatus is provided, comprising: acquiring scene information on a user’s current photographing; acquiring, based on the scene information, an associated picture associated with the scene information from historical pictures photographed by the user; and providing an operation instruction based on an acquisition result of the associated picture.

The scene information includes at least one of: position information on where the current photographing is performed, picture information on the current photographing, date information on when the current photographing is performed, or voice information of the user at the time of photographing;

The associated picture is from at least one of the following: a locally stored picture, a picture in a cloud album connected to the image capturing apparatus, or a picture stored in another apparatus associated with the image capturing apparatus.

If the associated picture exists, providing a corresponding photographing prompt to the user for the current photographing based on the associated picture; and/or, if no associated picture exists, prompting the user to directly perform the current photographing and using a currently photographed picture as a template picture, or prompting the user to directly perform the current photographing and select at least one picture from a plurality of pictures currently photographed as a template picture.

Information that a date of the current photographing is a holiday and/or anniversary, the picture information is a preview picture identified by the image capturing apparatus at the time of the current photographing, and the voice information of the user at the time of photographing comprises voice information that an intention of the current photographing is to photograph a souvenir picture.

The associated picture includes: a picture, position information of which matches, within a certain range, the position information in the scene information, where the current photographing is performed by the user; a picture, a picture information of which matches the picture information in the scene information to a certain extent; a picture, date information of which matches the date information in the scene information; or a picture, content information of which matches recognized voice information of the user at the time of photographing, or a picture identified by an artificial intelligence algorithm;

The acquiring, based on the scene information, an associated picture associated with the scene information from historical pictures photographed by the user comprises: acquiring the associated picture in response to the scene information satisfying at least one of following plurality of triggering conditions: there exists a historical picture photographed within a predetermined range of a current photographing position, a current photographing position is within a predetermined range of a famous attraction marked in a server connected to the image capturing apparatus, a date of the current photographing belongs to a holiday and/or anniversary, there exists a historically photographed picture comprising a currently photographed scenery, a currently photographed character is marked as a member of a souvenir picture, a number of members marked as members of a souvenir picture in currently photographed characters exceeds a predetermined number, a probability that a preview picture identified by the image capturing apparatus being a souvenir picture exceeds a predetermined value, or the voice information of the user at the time of photographing comprises an intention to photograph a souvenir picture;

The providing, based on the associated picture, a corresponding photographing prompt to the user for the current photographing comprise: displaying a list of thumbnails comprising the associated picture on a preview picture of the current photographing; and providing the photographing prompt based on the selected picture in response to the user selecting a picture from the list;

It adds labelling information to the photographed picture, after the user photographs a picture according to the photographing prompt.

It prompts the user to mark the photographed picture as a template picture, after the user photographs a picture according to the photographing prompt.

It displays a coincidence degree of the photographed picture and the selected picture on a preview picture, after the user photographing a picture according to the photographing prompt.

It prompting the user to set a reminder for asking whether to photograph a souvenir picture every predetermined interval, after the user photographs a picture according to the photographing prompt.

It synthesizes the photographed picture and the selected picture into a contrast picture, after the user photographs a picture according to the photographing prompt.

If no associated picture exists, it adds labelling information to the photographed template picture and/or sets a reminder to prompt the user to ask the user whether to photograph a souvenir picture every predetermined interval.

The acquiring the associated picture in response to the scene information satisfying the at least one triggering condition include: displaying a souvenir picture icon on the preview picture of the current photographing, in response to the scene information satisfying the at least one triggering condition; and acquiring the associated picture, in response to the user clicking the souvenir picture icon.

The identifying a probability that a preview picture of the image capturing apparatus is a souvenir picture comprises: identifying, by a statistical method or an artificial intelligence algorithm, the probability that the preview picture is the souvenir picture.

It turns on or off at least one of the plurality of triggering conditions.

The operations of displaying a list of thumbnails including the associated picture on a preview picture of the current photographing include: if a template picture exists in the associated picture, arranging a thumbnail of the template picture in front of a thumbnail of a non-template picture in the associated picture in the list; and if a template picture does not exist in the associated picture, determining a number of the associated picture, and displaying a list of thumbnails including all of the associated picture if the number of the associated picture does not exceed a predetermined value, and prompting the user to photograph a picture as a sample picture if the number of the associated picture exceeds the predetermined value, and matching the photographed sample picture with each of the associated picture, and preferentially displaying a thumbnail of the associated picture having a high matching degree.

A operation of the providing the photographing prompt based on the selected picture, in response to the user selecting a picture from the list includes: acquiring a feature content of the selected picture, and overlapping-displaying an outline of the feature content and/or text information about the feature content on the preview picture.

Acquiring the predetermined interval by photographing information.

The famous attraction includes at least one of a location a location where historical pictures are photographed and the number of the photographed pictures marked by the user as souvenir pictures reaches a certain threshold, a location marked by a database by default, or a popular tourist attraction.

The displaying a souvenir picture icon on the preview picture includes: displaying indication information for prompting the user to enter a souvenir picture mode around the souvenir picture icon.

The adding labelling information to the photographed picture includes: adding the labelling information to the photographed picture based on a difference between photographing information on the selected picture and photographing information on the photographed picture.

If a template picture is included in the associated picture, the original template picture is set as non-template picture in response to the user selecting to mark the photographed picture as the template picture.

The identifying, by an artificial intelligence algorithm, the probability that the preview picture is the souvenir picture includes: using a neural network model to identify the probability that the preview picture is the souvenir picture.

The synthesizing the photographed picture and the selected picture into a contrast picture includes: adding photographing information corresponding to the photographed picture and the selected picture, to the synthesized contrast picture.

The photographing information includes at least one of: position information on where the picture was photographed, date information on when the picture was photographed, information on whether a date of the photographed picture belongs to a holiday and/or anniversary, scenery information in the photographed picture, character information in the photographed picture, activity content information in the photographed picture, remark information on the selected picture, or title information on the selected picture.

The difference includes at least one of: a difference in photographing time, a difference in photographing position, a difference in a number and/or movement of photographed characters, or a difference in a photographed scenery.

The neural network model is obtained by iterative training with a set of pictures marked as souvenir pictures and a set of pictures not marked as souvenir pictures.

According to another aspect of the present disclosure, an image capturing apparatus is provided, including: a display; a processor, configured to: acquire scene information on a user’s current photographing; acquire an associated picture associated with the scene information from historical pictures photographed by the user based on the scene information; and provide an operation instruction based on an acquisition result of the associated picture.

The scene information includes at least one of position information, picture information, date information, or voice information of the user when photographing of the current photographing.

The processor is further configured to: if the associated picture exists, provide a corresponding photographing prompt for the user for the current photographing based on the associated picture; and/or, if no associated picture exists, prompt the user to directly perform the current photographing and use a currently photographed picture as a template picture, or prompt the user to directly perform the current photographing and select at least one picture from a plurality of pictures currently photographed as a template picture.

The date information includes information that a date of the current photographing is a holiday and/or anniversary, the picture information is a preview picture identified by the image capturing apparatus at the current photographing, and the voice information of the user when photographing includes voice information that an intention of the current photographing is to photograph a souvenir picture.

The associated picture includes at least one of the following: a picture, position information of which matches, within a certain range, the position information in the scene information, where the current photographing is performed by the user; a picture, a picture information of which matches the picture information in the scene information to a certain extent; a picture, date information of which matches the date information in the scene information; or a picture, content information of which matches the user’s recognized voice information at the time of photographing, or a picture identified by an artificial intelligence algorithm;

The processor is further configured to: acquire the associated picture in response to the scene information satisfying at least one of the following plurality of triggering conditions: a historical picture photographed within a predetermined range of a current photographing position exists, a current photographing position is within a predetermined range of a famous attraction marked in a server connected to the image capturing apparatus, a date of the current photographing belongs to a holiday and/or anniversary, a historically photographed picture including a currently photographed scenery exists, a currently photographed character is marked as a member of a souvenir picture, a number of members marked as members of a souvenir picture in currently photographed characters exceeds a predetermined number, a probability that a preview picture identified by the image capturing apparatus being a souvenir picture exceeds a predetermined value, or the voice information of the user when photographing includes an intention to photograph a souvenir picture.

The processor is further configured to: display a list of thumbnails including the associated picture on a preview picture of the current photographing; and provide the photographing prompt based on the selected picture, in response to the user selecting a picture from the list.

The processor is further configured to: add labelling information to the photographed picture, after the user photographs a picture according to the photographing prompt.

The processor is further configured to: prompt the user to mark the photographed picture as a template picture, after the user photographs a picture according to the photographing prompt.

The processor is further configured to: display a coincidence degree of the photographed picture and the selected picture on a preview picture, after the user photographing a picture according to the photographing prompt.

The processor is further configured to: prompt the user to set a reminder for asking the user whether to photograph a souvenir picture every predetermined interval, after the user photographing a picture according to the photographing prompt.

The processor is further configured to: synthesize the photographed picture and the selected picture into a contrast picture, after the user photographing a picture according to the photographing prompt.

The processor is further configured to: if no associated picture exists, add labelling information to the photographed template picture and/or set a reminder to prompt the user to ask the user whether to photograph a souvenir picture every predetermined interval.

The processor is further configured to: display a souvenir picture icon on the preview picture of the current photographing, in response to the scene information satisfying the at least one triggering condition; and acquire the associated picture, in response to the user clicking the souvenir picture icon.

The processor is further configured to: identify, by a statistical method or an artificial intelligence algorithm, the probability that the preview picture is the souvenir picture.

The processor is further configured to: turn on or off at least one of the plurality of triggering conditions.

The processor is further configured to: if a template picture exists in the associated picture, arrange a thumbnail of the template picture in front of a thumbnail of a non-template picture in the associated picture in the list; and if a template picture does not exist in the associated picture, determine a number of the associated picture, and display a list of thumbnails including all of the associated picture if the number of the associated picture does not exceed a predetermined value, and prompt the user to photograph a picture as a sample picture if the number of the associated picture exceeds the predetermined value, and match the photographed sample picture with each of the associated picture, and preferentially display a thumbnail of the associated picture having a high matching degree.

The processor is further configured to: acquire a feature content of the selected picture, and overlapping-display an outline of the feature content and/or text information about the feature content on the preview picture.

The processor is further configured to: acquire the predetermined interval by photographing information.

The famous attraction includes at least one of a location where historical pictures are photographed and the number of the photographed pictures marked by the user as souvenir pictures reaches a certain threshold,, a location marked by a database by default, or a popular tourist attraction.

The processor is further configured to: display indication information for prompting the user to enter a souvenir picture mode around the souvenir picture icon.

The processor is further configured to: add the labelling information to the photographed picture based on a difference between photographing information on the selected picture and photographing information on the photographed picture.

The processor is further configured to: if a template picture is included in the associated picture, set the original template picture as non-template picture in response to the user selecting to mark the photographed picture as the template picture.

The processor is further configured to: use a neural network model to identify the probability that the preview picture is the souvenir picture.

The processor is further configured to: add photographing information corresponding to the photographed picture and the selected picture, to the synthesized contrast picture. The photographing information includes at least one of: position information on where the picture is photographed, date information on when the picture is photographed, information on whether a date of the photographed picture belongs to a holiday and/or anniversary, scenery information in the photographed picture, character information in the photographed picture, activity content information in the photographed picture, remark information on the selected picture, or title information on the selected picture.

The neural network model is obtained by a iterative training process with a set of pictures marked as souvenir pictures and a set of pictures not marked as souvenir pictures.

According to another aspect of the present disclosure, a computer readable storage medium is provided, storing a program including a code for implementing the auxiliary photographing method according to any one of the above embodiments.

According to another aspect of the present disclosure, a computer is provided, including a readable medium storing a computer program, the computer program including a code for implementing the auxiliary photographing method according to any one of the above embodiments.

An aspect of the present disclosure is to provide an auxiliary photographing method and an image capturing apparatus that may photograph a picture that is highly coincident with a historically photographed picture.

The above and other objectives as well as features of embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings illustratively showing the embodiments, in which:

Fig. 1 is a flowchart of an auxiliary photographing method according to an embodiment of the present disclosure;

Fig. 2 illustrates a setting interface of a souvenir picture icon according to an embodiment of the present disclosure;

Figs. 3a-3h are schematic diagrams of auxiliary photographing by an image capturing apparatus according to the embodiments of the present disclosure; and

Fig. 4 is a block diagram of the image capturing apparatus according to the embodiments of the present disclosure.

Reference will now be made in detail to embodiments of the present disclosure. Examples of the embodiments are shown in the accompanying drawings. Here, the same reference numerals indicate the same elements all through this disclosure. The embodiments are described below in order to explain the present disclosure with reference to the accompanying drawings.

When a user uses an image capturing apparatus to photograph a souvenir picture, by observing the difference between the character and background in a historically photographed picture and the character and background of a currently photographing with naked eyes, without prompting, it is often impossible to obtain a souvenir picture that is highly similar to the historically photographed picture. In the present disclosure, an auxiliary photographing method and an image capturing apparatus are proposed, to provide a photographing prompt using a historically photographed picture to enable the user to photograph a souvenir picture that is highly similar to the historically photographed picture.

Fig. 1 is a flowchart of an auxiliary photographing method according to an embodiment of the present disclosure.

In operation 101, scene information on a user's current photographing is acquired.

According to an embodiment of the present disclosure, when the user prepares to photograph using the image capturing apparatus, scene information on the current photographing may be acquired. Here, the image capturing apparatus may be a camera dedicated to photographing, or may be a device having a camera such as a smartphone. The following description is made by taking a smart terminal device (smartphone) having a touch picture as an example, but those skilled in the art may easily understand that the auxiliary photographing method according to the embodiments of the present disclosure may be applied to other devices having an image capturing function. According to an embodiment of the present disclosure, the scene information may include at least one of position information, picture information, date information, or voice information of the user at the time of the current photographing. Here, the date information includes information that the date of the current photographing is a holiday and/or anniversary, the picture information is a preview picture identified by the image capturing apparatus at the time of the current photographing, and the voice information of the user at the time of photographing includes voice information that the intention of the current photographing is to photograph a souvenir picture. For example, the acquired scene information on the current photographing is "Xuanwu Lake Park, No. 1, Xuanwu District, Nanjing (118.76, 32.04)", or the acquired scene information is "March 10, 2018, wedding anniversary". It should be understood that the above examples of the scene information are merely examples, and any information related to the photographed scene may be included in the scene information according to the embodiments of the present disclosure.

Specifically, the image capturing apparatus may: acquire the position information on where the current photographing is performed, by reading the current GPS position; acquire the date information on when the current photographing is performed, by reading the current date; and determine whether the date of the current photographing belongs to a holiday specified in the calendar in the image capturing apparatus or specified in a server connected to the image capturing apparatus, or an anniversary pre-set by the user in the image capturing apparatus. In addition, the image capturing apparatus may also acquire the scene information and the character information on the current photographing by the preview picture of the current photographing. In addition, the image capturing apparatus may also acquire the voice information of the user at the time of photographing, by using a microphone provided in the image capturing apparatus.

In operation 102, an associated picture associated with the scene information is acquired from historical pictures photographed by the user, based on the scene information.

According to an embodiment of the present disclosure, the associated picture is acquired from the historical pictures photographed by the user, based on the acquired scene information. The historical pictures are generally stored in a local memory, in a cloud album connected to the image capturing apparatus, or in other storage apparatuses associated with the image capturing apparatus, then the acquired picture associated with the scene information is also obtained from locally stored pictures, pictures in the cloud album connected to the image capturing apparatus, or pictures stored in the other apparatuses associated with the image capturing apparatus. Specifically, the associated picture may be acquired in response to the scene information satisfying at least one of the plurality of triggering conditions that may include: there exists a historical picture photographed within a predetermined range of a current photographing position, the current photographing position is within a predetermined range of a famous attraction marked in the server connected to the image capturing apparatus, the date of the current photographing belongs to a holiday and/or anniversary, there exists a historically photographed picture including a currently photographed scenery, a currently photographed character is marked as a member of a souvenir picture, a number of members marked as members of a souvenir picture in currently photographed characters exceeds a predetermined number, a probability that a preview picture identified by the image capturing apparatus being a souvenir picture exceeds a predetermined value, or the voice information of the user at the time of photographing includes an intention to photograph a souvenir picture. It should be understood that, with local and/or remote assistance, the image capturing apparatus according to the embodiments of the present disclosure may determine whether a triggering condition is satisfied. For example, whether a triggering condition is satisfied may be determined by starting a process in the image capturing apparatus, running an artificial intelligence (AI) system on the image capturing apparatus, or through the server connected to the image capturing apparatus. Here, the associated picture includes: a picture, position information of which matches, within a certain range, the position information in the scene information, where the current photographing is performed by the user; a picture, a picture information of which matches the picture information in the scene information to a certain extent; a picture, date information of which matches the date information in the scene information; or a picture, content information of which matches recognized voice information of the user, which is obtained at the time of photographing, or a picture identified by an artificial intelligence algorithm;

According to an embodiment of the present disclosure, a souvenir picture icon may be displayed on the preview picture of the current photographing, in response to the scene information satisfying at least one triggering condition of the plurality of triggering conditions, and the associated picture may be acquired in response to the user clicking the souvenir picture icon. According to an embodiment of the present disclosure, when displaying the souvenir picture icon, indication information for prompting the user to enter a souvenir picture mode may be displayed around the souvenir picture icon, such as "click to enter the souvenir picture mode" or "souvenir picture mode, click to view available templates". If the user does not click the souvenir picture icon within a predetermined time (for example, 5 seconds) after the souvenir picture icon is displayed, then it stops displaying the souvenir picture icon, or the souvenir picture icon may be displayed until the scene information does not satisfy the triggering condition and then disappear (i.e., stop displaying the souvenir picture icon). According to an embodiment of the present disclosure, the souvenir picture icon that has been stopped to be displayed may be further displayed when the scene information in the current photographing is re-satisfied with the triggering condition, or until the next time the camera is turned on for a new photographing, the souvenir picture icon is displayed again based on the triggering condition being satisfied. For example, when the image capturing apparatus that performs photographing is located at position A, if a historical picture photographed within a predetermined range of the position A exists in the image capturing apparatus, it is determined that the triggering condition is satisfied. Here, the user may set the predetermined range satisfying the triggering condition (for example, setting a predetermined range of 20 meters) in the image capturing apparatus. In addition, if the user marks the locations of some famous attractions in a server cloud connected to the image capturing apparatus, or the user reports location information to the server when photographing a picture, and when the number of times of the reported location information being marked as a famous attraction reaches a certain value, the server may mark the location as a “famous attraction”, or some famous attractions set by the system by default and some famous attractions that the system may intelligently identify. Here, the famous attractions set by the system by default may be stored in a local database, and the famous attractions that the system intelligently identifies are well-known attractions, such as the Forbidden City, the Great Wall, and Qingcheng Mountain. In this case, if the location A is around a famous attraction (for example, the Forbidden City) marked in the server, the triggering condition is satisfied. Further, if the date when the current photographing is performed belongs to a holiday (for example, the Mid-Autumn Day) or an anniversary (for example, wedding anniversary) pre-set by the user in the image capturing apparatus, it is determined that the triggering condition is satisfied. Further, if the preview picture of the image capturing apparatus currently performing photographing includes a building and/or a landscape (for example, Tiananmen), if a historically photographed picture including the scenery on the preview image exists in the image capturing apparatus, for example, a picture photographed one year ago includes Tiananmen, it is determined that the triggering condition is satisfied. Further, for example, the image capturing apparatus may turn on a process when an image is captured, so that target detection classification of the bitmap of the preview picture is performed every 1 second by using, for example, a convolutional neural network algorithm. When it is detected that there is a frontal face on the preview picture, it is determined that the triggering condition is satisfied. Further, for example, if the user has previously edited the characters existing in the historically photographed pictures in the image capturing apparatus, and marked the label "souvenir picture member", then, when the user performs photographing, a face recognition system of the image capturing apparatus may recognize faces on the preview picture by using a face recognition algorithm, and when the number of recognized souvenir picture members appearing on the preview picture exceeds, for example, three, it is determined that the triggering condition is satisfied. Further, for example, the image capturing apparatus may identify the probability that the preview picture is a souvenir picture by a statistical method or an AI system. For example, the image capturing apparatus may compare the character costumes on the preview picture with the costumes of the characters in the historically photographed pictures, and when the degree of similarity reaches a predetermined value, it is determined that the preview picture is a souvenir picture, and it is determined that the triggering condition is satisfied. It should be understood that the above-described examples regarding satisfying the triggering condition are merely exemplary examples, and examples of the triggering condition being satisfied that may be used in the present disclosure are not limited thereto.

According to an embodiment of the present disclosure, the AI system may perform neural network iterative training by using a set of pictures marked as souvenir pictures and a set of pictures not marked as souvenir pictures, and use the trained neural network to identify the probability that the preview picture is a souvenir picture. First, training samples are selected. The selected training samples include two sets of training pictures, one set of souvenir pictures and the other of non-souvenir pictures. The souvenir picture is usually a picture including one or more frontal/side characters in the foreground and the background is some landmark buildings or natural scenery highly recognizable or the like, and the non-souvenir picture is usually a picture of still life, scenery without characters, animal, food or the like. Then, the selected training samples are subjected to data preprocessing including size adjustment, noise elimination, normalization, etc., and the two sets of pictures are respectively marked with labels "souvenir picture" and "non-souvenir picture". Image data obtained after the above data preprocessing is used as an input, the labels correspondingly marked are used as output, and the neural network is iteratively trained based on a deep learning framework to construct a neural network model. After a neural network is trained, the user does not need to manually adjust or set parameters, instead the user only needs to use a picture as an input of the neural network, and the probability that the picture is identified as a souvenir picture may be output. In this case, when it is identified the probability that the preview picture is a souvenir picture exceeds a predetermined value, it is determined that the triggering condition is satisfied. Here, common deep learning frameworks include TensorFlow, Caffe, Theano, Keras, PyTorch, MXNet, or the like. Deep learning frameworks are commonly used in fields such as computer vision, voice recognition, natural language processing, or bioinformatics. The neural network model may be constructed using convolutional neural networks (CNN), recurrent neural networks (RNN), a long short term memory network (LSTM) or the like.

According to an embodiment of the present disclosure, if a voice recognition system of the image capturing apparatus recognizes that the voice of the user at the time of photographing includes an intention to photograph a souvenir picture, for example, the user says "photograph a souvenir picture for me", it is determined that the triggering condition is satisfied. Alternatively, the user says "the scenery here is beautiful", "we photographed here last year", "we should photograph here every year", etc., the voice recognition system may ask if the user needs to photograph a souvenir picture, if the user answers "Yes", then it is determined that the triggering condition is satisfied.

According to an embodiment of the present disclosure, the user may turn on or off at least one of the plurality of triggering conditions described above. Fig. 2 illustrates a setting interface of a souvenir picture icon according to an embodiment of the present disclosure. As shown in Fig. 2, a camera setting interface (210) may display a setting option (211) of the souvenir picture icon, and the user may set whether to display the souvenir picture icon in the interface, that is, set whether to provide a photographing prompt. When the user sets to turn off, the souvenir picture icon is not displayed when using the image capturing apparatus to photograph pictures, that is, no photographing prompt is provided. When the user sets to turn on, the photographing prompt is to be provided according to the method shown in Fig. 1. The user may click the setting option of the souvenir picture icon to enter the setting interface of the souvenir picture icon (230) . In the setting interface of the souvenir picture icon (230), the user may set to turn on and off of each of the triggering conditions. When the user sets to turn off a specific triggering condition, the souvenir picture icon is not displayed even if the triggering condition is satisfied, and thus the associated picture is not acquired when the triggering condition is satisfied.

Referring back to Fig. 1, in operation 103, an operation instruction is provided based on an acquisition result of the associated picture.

According to an embodiment of the present disclosure, a list of thumbnails including the associated picture may be displayed on a preview picture of the current photographing. In response to the user selecting a picture from the list, a photographing prompt is provided based on the selected picture, or the system selects a picture by default, and a photographing prompt is provided based on the picture selected by the system by default.

According to an embodiment of the present disclosure, it is determined whether a template picture exists in the associated picture. If a template picture exists in the associated picture, a thumbnail of the template picture is arranged in front of a thumbnail of a non-template picture in the associated picture in the list. If a template picture does not exist in the associated picture, the number of the associated picture is determined. A list of thumbnails including all of the associated picture is displayed, if the number of the associated picture does not exceed a predetermined value. The user is prompted to photograph a picture as a sample picture if the number of the associated picture exceeds the predetermined value, and then a degree of matching between the photographed sample picture and each of the associated picture is determined, and a thumbnail of the associated picture having a high matching degree is preferentially displayed. Or, the associated pictures are searched directly to obtain the associated picture having a high matching degree with the currently acquired preview picture, and the thumbnail of the associated picture having the high matching degree is preferentially displayed. The user may arbitrarily select an associated picture as the template picture from the thumbnails of the plurality of associated pictures displayed. For example, if the associated picture includes a template picture, then in the list, the template picture may be displayed in front of a non-template picture, and the template picture may be marked with a corner mark, etc. to be distinguished from the non-template picture. If the associated picture includes a plurality of pictures of different backgrounds, in order to facilitate the user’s selection, pictures belonging to the same background may be discriminated by a feature point extraction and matching method (for example, ORB, SIFT, histogram, etc.), and only one of the plurality of pictures of the same background is displayed. If there is a template picture in the plurality of pictures of the same background, only the template picture is displayed. If there is no template picture in the plurality of pictures of the same background, a best one may be selected and displayed by judging the pros and cons of the pictures. As another example, if the number of associated picture exceeds 10, and the associated picture cannot be all displayed in the list on the preview picture, then the user is prompted to photograph a picture as the sample picture, and after the photographing, a process is started to perform "searching by picture" in the background of the image capturing apparatus. For example, first, these pictures, which are photographed near the position where the sample picture was photographed, are searched in the associated picture, to obtain a set of pictures historically photographed near the current position, and then the feature point extraction method (for example, an algorithm such as ORB, SIFT, or histogram) is used to perform feature point extraction on the sample picture and all the pictures in the set, to obtain feature point descriptors. After the feature points are extracted, a matching algorithm (for example, a k-proximity classification algorithm)is used to determine a matching degree between the feature point descriptor of the sample picture and the feature point descriptor of each picture in the set. The more the number of feature points successfully are matched, the higher the similarity of the two pictures is. The pictures in the set are arranged in descending order in the list based on the number of feature points that the sample picture successfully matches with each picture in the set. However, there is a difference in the scenery in pictures photographed in different seasons, such as difference in color, or tree clipping condition. In this case, the grayscale images may be discriminated by a feature point matching algorithm, thereby ignoring the difference in color. At the same time, in order to improve the matching efficiency, the matching process may be performed on the thumbnails of the pictures to ignore a small amount of content change. In addition, it may directly matching-search the associated pictures based on the currently acquired preview picture, to search out the associated picture having high matching degree with the currently acquired preview picture, and the thumbnail of the associated picture having the high matching degree is preferentially displayed. The above description is merely exemplary, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, after the user selects a picture from the list, the feature content of the selected picture may be acquired, and an outline of the feature content and/or text information about the feature content are displayed as overlapped on the preview picture, so that the user may photograph a picture based on the outline or the text information overlapping-displayed. For example, the image capturing apparatus may extract the outline by using an existing image processing method, such as first performing grayscale processing on the picture, and then extracting edges using an image gradient algorithm. The image capturing apparatus may overlapping-display the background outline, the character movement outline, and the like of the selected picture in a watermark-like form on the preview picture, or may overlapping display a text on the preview picture to mark the position of each character. Therefore, the user may adjust the position and angle of the camera based on the background outline, adjust the movement of the character to be photographed based on the movement outline, adjust the stance of the character to be photographed based on the text, or the like, thereby photographing a picture having a high degree of coincidence with the selected picture.

According to an embodiment of the present disclosure, after photographing a picture according to the photographing prompt, the user may add labelling information to the photographed picture. Here, the labelling information may be added to the photographed picture based on the difference between the photographing information on the selected picture and the photographing information on the photographed picture. Here, the photographing information includes at least one of: position information on where the photographed picture is obtained, date information on when the photographed picture is obtained, information on whether the date of the photographed picture belongs to a holiday and/or anniversary, scenery information in the photographed picture, character information in the photographed picture, activity content information in the photographed picture, remark information on the selected picture, or title information on the selected picture. The difference in photographing may be a difference in photographing time of the selected picture and photographing time of the photographed picture, a difference in photographing position, a difference in a photographed scenery, or a difference in the number and/or movement of photographed characters. For example, if the user selects to provide a photographing prompt for a picture photographed 2 years ago stored in the image capturing apparatus having a remark of souvenir picture of the first wedding anniversary, and photographs a new picture according to the photographing prompt, labelling information "souvenir picture of the third wedding anniversary" may be added to the newly photographed picture based on the difference in the photographing time. According to an embodiment of the present disclosure, the labelling information may be title information or remark information on the picture, or explanation information added to the picture.

According to an embodiment of the present disclosure, the photographed picture and the selected picture may be synthesized into a contrast picture, after the user photographs a picture according to the photographing prompt. Also, photographing information corresponding to the photographed picture and the selected picture may be added to the synthesized contrast picture. For example, a picture photographed 2 years ago having a remark of souvenir picture of the first wedding anniversary selected by the user and a souvenir picture of the third wedding anniversary photographed based on the souvenir picture of the first wedding anniversary may be synthesized into one picture. In the synthesized picture, on the left side is the souvenir picture of the first wedding anniversary, and on the right side is the souvenir picture of the third wedding anniversary. In addition, photographing information such as photographing year or photographing position may be added below the souvenir picture of the first wedding anniversary and the souvenir picture of the third wedding anniversary in the synthesized picture. However, the above examples are merely exemplary, and the present invention is not limited thereto. However, the above example is merely exemplary, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, after the user photographs a picture according to the photographing prompt, the user may be prompted to mark the photographed picture as the template picture. If a template picture is included in the associated picture, the original template picture may be set as non-template picture in response to the user setting the photographed picture as the template picture. After the user photographs a picture according to the photographing prompt, a coincidence degree of the photographed picture and the selected picture may be displayed on the preview picture. For example, after a picture is photographed, the background outline, the character movement outline, or the like in the newly photographed picture may be extracted, the coincidence degree of the corresponding outline in the new and old pictures is calculated, and the calculated coincidence degree is displayed on the preview picture. After the user photographs a picture according to the photographing prompt, the user may also be prompted to set a reminder for asking the user whether to photograph a souvenir picture every predetermined interval. For example, after the user has photographed a souvenir picture according to the photographing prompt, a prompt message "Whether to set to photograph a souvenir picture in this background every other year?" may be displayed on the preview picture. Here, the predetermined interval may be set based on the remark information, the aforementioned photographing interval of the souvenir picture, time information, and the like, such as setting the predetermined interval of a picture of the remark information "wedding anniversary" to 1 year, setting the predetermined interval of a picture of the title information "quarterly event" to 3 months, setting the predetermined interval of a picture of the remark information "Christmas" to 1 year, or setting the fitness management picture to remind by month or by week. It should be understood that the above examples are merely exemplary and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, if the associated picture is not acquired in operation 102, a photographing prompt cannot be provided for the current photographing. In this case, the user is prompted to directly perform the current photographing and using a currently photographed picture as the template picture, or the user is prompted to directly perform the current photographing and select at least one picture from a plurality of pictures currently photographed as the template picture. In addition, it is also possible to add labelling information to the photographed template picture and/or set a reminder to prompt the user whether to photograph a souvenir picture every predetermined interval.

According to the above auxiliary photographing method, a photographing prompt may be provided based on a historically photographed picture, so that the user may photograph a picture having a high degree of coincidence with the historically photographed picture.

Figs. 3a-3h are schematic diagrams of auxiliary photographing by an image capturing apparatus according to the embodiments of the present disclosure.

As shown in Fig. 3(a), in response to the scene information on the current photographing satisfying at least one triggering condition, a souvenir picture icon shown by the square on the picture in Fig. 3(a) is displayed on the preview picture of the current photographing. At the same time, in the form of a pop-up window, indication information for prompting the user to enter a souvenir picture mode is displayed around the souvenir picture icon. In addition to displaying the indication information in the form of a pop-up window as shown in Fig. 3(a), the indication information may also be displayed in a floating message, a text, an icon, a voice, or the like, and the displayed indication information disappears after a certain time (for example, 2 seconds). In addition, the souvenir picture icon may be different depending on the picture on the preview picture. For example, if the picture on the preview picture is a landscape, the souvenir picture icon may be displayed as an icon representing landscape instead of the square as shown in Fig. 3(a). If the picture on the preview picture is flowers and plants, the souvenir picture icon may be displayed as an icon representing flowers and plants. If the picture on the preview picture is a building, the souvenir picture icon may be displayed as an icon representing building. The above description is merely exemplary, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, in response to the user clicking the souvenir picture icon within the predetermined time (e.g., 5 seconds), the associated picture associated with the scene information is acquired, and a list of thumbnails including the associated picture is displayed. As shown in Figs. 3(b) and 3(c), the list is displayed vertically on the left side of the preview picture. In addition, the list may be displayed vertically on the right side of the preview picture, or horizontally above or below the preview picture. As shown in Fig. 3(b), if template pictures exist in the associated pictures, the template pictures are arranged in front of non-template pictures in the list, and the template pictures are marked with corner marks to be distinguished from the non-template pictures. As shown in Fig. 3(c), if a template picture does not exist in the associated pictures, and the associated pictures are too many, for example, the alternative souvenir pictures are more than 10, the user may be prompted to photograph a picture as a sample picture, to search for a picture similar to the sample picture in the associated pictures, and arrange the thumbnails of the associated pictures in the list in descending order according to the degree of matching similarity between the sample picture and each picture in the associated pictures after the search. In addition, if the number of the associated picture is less than or equal to 10, they may be sorted and displayed in the list according to the pros and cons of the composition of the picture, color, and the like. Alternatively, a picture identified by the AI system as a souvenir picture may also be preferentially displayed in the list.

According to an embodiment of the present disclosure, as shown in Fig. 3(d), in response to the user selecting a picture from the list, the background outline of the selected picture may be displayed on the preview picture in the form of a watermark, so that the user may adjust the position and angle of the camera based on the watermark to make the background to be photographed highly coincident with the watermark. After the user finishes photographing the picture based on the watermark of the background outline, the newly photographed picture may be compared with the selected picture, the coincidence degree of the two pictures is calculated, and the coincidence degree is displayed on the preview picture, as shown in Fig. 3(e). In addition, as shown in Fig. 3(f), after the user finishes photographing the picture based on the watermark of the background outline, the user may be prompted on the preview picture to set the photographed picture as a template picture. In addition to prompting the user to set a template picture in the form of a pop-up window, the user may also be prompted by displaying in the form of a floating message, text, icon, voice, etc., and the present disclosure is not limited thereto. In addition, the user may also set a picture as a template picture when browsing the pictures in the album, for example, as shown in Fig. 3(g), the user may set the picture as a template picture by clicking the menu icon on the upper right corner of the picture.

According to an embodiment of the present disclosure, when clicking the souvenir picture icon as shown in Fig. 3(a), if no associated picture is acquired, the user may be prompted to set the photographed picture as a template picture after the user photographs a picture, such as shown in Fig. 3(h). If the souvenir picture icon as shown in Fig. 3(a) is clicked and no existing template or picture photographed in the same location (i.e., associated picture) is acquired, the user may be prompted to set the photographed picture as a template picture after the user photographs a picture.

Referring to Fig. 4, the image capturing apparatus 400 may include a lens assembly 410, a flash 420, an image sensor 430, an image stabilizer 440, a memory 450 (e.g., buffer memory), a display 460, and/or a processor 470. Here, the image capturing apparatus 400 may be a camera dedicated to photographing, or may be a device having a camera such as a smartphone. The following description is made by taking a smart terminal device (smartphone) having a touch picture as an example, but those skilled in the art may easily understand that the auxiliary photographing method according to the embodiments of the present disclosure may be applied to other devices having an image capturing function.

The lens assembly 410 may collect light emitted or reflected from an object to be photographed. The lens assembly 410 may include one or more lenses. According to an embodiment, the image capturing apparatus 400 may include a plurality of lens assemblies 410. In this case, the image capturing apparatus 400 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 410 may have the same lens properties (e.g., viewing angle, focal length, auto focus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of the other lens assemblies. The lens assembly 410 may include, for example, a wide-angle lens or a telephoto lens.

The flash 420 may emit light, where the emitted light is used to enhance light reflected from an object. According to an embodiment, the flash 420 may include one or more light emitting diodes (LED) (e.g., red green blue (RGB) LED, white LED, infrared (IR) LED, or ultraviolet (UV) LED) or xenon lamps. The image sensor 430 may acquire an image corresponding to the object by converting light emitted or reflected from the object and transmitted through the lens assembly 410 into an electrical signal. According to an embodiment, the image sensor 430 may include one image sensor (e.g., RGB sensor, black and white (BW) sensor, IR sensor, or UV sensor) selected from a plurality of image sensors having different properties, a plurality of image sensors having the same property or a plurality of image sensors having different properties. Each of the image sensors included in the image sensor 430 may be implemented using, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 440 may move the image sensor 430 or at least one lens included in the lens assembly 410 along a particular direction, or control the operational properties of the image sensor 430 (e.g., adjust the readout timing) in response to a movement of the image capturing apparatus 400. In this way, it is allowed to compensate for at least a portion of negative effects (e.g., image blur) resulting from the movement of an image being captured. According to an embodiment, the image stabilizer 440 may sense such movement of the image capturing apparatus 400 using a gyro sensor (not shown) or an acceleration sensor (not shown) arranged within or outside of the image capturing apparatus 400. According to an embodiment, the image stabilizer 440 may be implemented as, for example, an optical image stabilizer.

The memory 450 may at least temporarily store at least a portion of images acquired via the image sensor 430 for subsequent image processing tasks. For example, if a plurality of images are quickly captured or image capturing is delayed due to shutter lag, the acquired raw images (e.g., Bayer pattern images, high resolution images) may be stored in the memory 450 and their corresponding copy images (for example, low resolution images) may be reviewed via the display 460. Then, if a specified condition is met (e.g., by the user's input or system command), at least a portion of the raw images stored in the memory 450 may be acquired and processed by, for example, the processor 470.

The display 460 may display a photographing preview picture when the image capturing apparatus 400 is in photographing. In addition, the display 460 may also display various user interfaces according to the user's operation, such as an album user interface, or a user interface for setting the image capturing apparatus.

The processor 470 may perform one or more image processing on an image acquired via the image sensor 430 or an image stored in the memory 450. Here, the one or more image processing may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, picture synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blur, sharpen or soften). Additionally or alternatively, the processor 470 may control (e.g., exposure time control or readout timing control) at least one of the components included in the image capturing apparatus 400 (e.g., the image sensor 430). The image processed by the processor 470 may be stored back to the memory 450 for further processing, or the image may be provided to an external component outside of the image capturing apparatus 400.

The processor 470 may also control the image capturing apparatus 400 to perform the auxiliary photographing method.

Specifically, the processor 470 may acquire scene information on a user’s current photographing.

According to an embodiment of the present disclosure, when the user prepares to photograph using the image capturing apparatus 400, the scene information on the current photographing may be acquired by the processor 470. According to an embodiment of the present disclosure, the acquired scene information may include at least one of position information on where the current photographing is performed, date information on when the current photographing is performed, information on whether the date of the current photographing belongs to a holiday and/or anniversary, scenery information on the current photographing, character information on the current photographing, or voice information of the user at time of photographing. It should be understood that the above examples of the scene information are merely examples, and any information related to the photographed scene may be included in the scene information according to the embodiments of the present disclosure.

Specifically, the processor 470 may acquire the position information on where the current photographing is performed, by reading the current GPS position, acquire the date information on when the current photographing is performed, by reading the current date, and determine whether the date of the currently performed photographing belongs to a holiday specified in the calendar in the image capturing apparatus 400 or specified in a server connected to the image capturing apparatus 400, or an anniversary pre-set by the user in the image capturing apparatus 400. In addition, the processor 470 may also acquire the scene information and the character information on the currently performed photographing through the preview picture of the currently performed photographing. In addition, the processor 470 may also acquire the voice information of the user at time of photographing by using a microphone provided in the image capturing apparatus 400.

The processor 470 may acquire an associated picture associated with the scene information from historical pictures photographed by the user based on the scene information.

According to an embodiment of the present disclosure, the processor 470 may acquire the associated picture associated with the scene information historically photographed based on the scene information from pictures locally stored in the image capturing apparatus 400, pictures in a cloud album connected to the image capturing apparatus 400, or pictures stored in the other apparatuses associated with the image capturing apparatus 400.

According to an embodiment of the present disclosure, the processor 470 may acquire the associated picture in response to the scene information satisfying at least one of the following plurality of triggering conditions. For example, the triggering condition may include: a historical picture photographed within a predetermined range of a current photographing position exists, the current photographing position is around a famous attraction marked in the server connected to the image capturing apparatus 400, the date of the current photographing belongs to a holiday and/or anniversary, a historically photographed picture including a currently photographed scenery exists, a currently photographed character is marked as a member of a souvenir picture, a number of members marked as members of a souvenir picture in currently photographed characters exceeds a predetermined number, a probability that a preview picture being a souvenir picture identified by the processor 470 exceeds a predetermined value, or the voice information on the user when photographing includes an intention to photograph a souvenir picture. It should be understood that the image capturing apparatus 400 according to the embodiments of the present disclosure may determine whether a triggering condition is satisfied by local and/or remote assistance. For example, the processor 470 may determine whether a triggering condition is satisfied by starting a process in the image capturing apparatus 400, running an artificial intelligence (AI) system on the image capturing apparatus 400, or through the server connected to the image capturing apparatus 400.

According to an embodiment of the present disclosure, the processor 470 may display a souvenir picture icon on the preview picture of the current photographing, in response to the scene information satisfying at least one triggering condition of the plurality of triggering conditions, and acquire the associated picture in response to the user clicking the souvenir picture icon. According to an embodiment of the present disclosure, when displaying the souvenir picture icon, indication information for prompting the user to enter a souvenir picture mode may be displayed around the souvenir picture icon. If the user does not click the souvenir picture icon within a predetermined time (for example, 5 seconds) after the souvenir picture icon is displayed, the souvenir picture icon stops being displayed, and before the user finishes photographing (i.e., exits the camera application), the souvenir picture icon is no longer displayed, i.e., no photographing prompt is provided, until the next time the camera is turned on for photographing, and the souvenir picture icon is displayed again based on the triggering condition being satisfied.

For example, when the image capturing apparatus 400 that performs photographing is located at position A, if a historical picture photographed within a predetermined range around the position A exists in the image capturing apparatus 400, it is determined that the triggering condition is satisfied. According to an embodiment of the present disclosure, the user may set the predetermined range satisfying the triggering condition (for example, 20 meters) in the image capturing apparatus 400. In addition, for example, if the locations of some famous attractions are marked in a server cloud connected to the image capturing apparatus 400, or the user reports location information to the server when photographing a picture, and when the number of times of a location being marked as a famous attraction reaches a certain value, the server may mark the location as a “famous attraction”. In this case, if the location A is around a famous attraction (for example, the Forbidden City) marked in the server, it is determined that the triggering condition is satisfied. Further, for example, if the date of the current photographing belongs to a holiday (for example, the Mid-Autumn Day) or an anniversary (for example, wedding anniversary) pre-set by the user in the image capturing apparatus 400, it is determined that the triggering condition is satisfied. Further, for example, if the preview picture of the image capturing apparatus 400 currently performing photographing includes a building and/or a landscape (for example, Tiananmen), if a historically photographed picture including the scenery on the preview image exists in the image capturing apparatus 400, for example, a picture photographed one year ago including Tiananmen, it is determined that the triggering condition is satisfied.

In addition, for example, the processor 470 may turn on a process when an image is captured, so that target detection classification of the bitmap of the preview picture is performed every 1 second by using, for example, a convolutional neural network algorithm. When it is detected that there is a frontal face on the preview picture, it is determined that the triggering condition is satisfied. Further, for example, if the user has previously edited the characters existing in the historically photographed pictures in the image capturing apparatus 400, and marked the label "souvenir picture member", then, when the user performs photographing, a face recognition system of the image capturing apparatus 400 may recognize faces on the preview picture by using a face recognition algorithm, and when the number of recognized souvenir picture members appearing on the preview picture exceeds, for example, three, it is determined that the triggering condition is satisfied. Further, for example, the processor 470 may identify the probability that the preview picture is a souvenir picture by a statistical method or an AI system. For example, the processor 470 may compare the character costumes on the preview picture with the costumes of the characters in the historically photographed pictures, and when the degree of similarity reaches a predetermined value, it is determined that the preview picture is a souvenir picture, and it is determined that the triggering condition is satisfied.

In addition, according to an embodiment of the present disclosure, the AI system may perform neural network iterative training using a set of pictures marked as souvenir pictures and a set of pictures not marked as souvenir pictures, and use the trained neural network to identify the probability that the preview picture is a souvenir picture. For example, two sets of training pictures may be prepared in advance, one set are of souvenir pictures. For example, the souvenir picture may be a picture including characters, and the background is some landmark buildings or highly recognizable plants, landscapes, etc., and the other set is other pictures being non-souvenir pictures, for example, pictures of still life, landscape without characters, etc. The labels "souvenir picture" and "non-souvenir picture" are marked for the two sets of pictures as desired outputs, and then the neural network is iteratively trained, and the trained network may be used to identify the probability that a picture is a souvenir picture. After a neural network is trained, the user does not need to manually adjust or set parameters, and only needs to use a picture as an input of the neural network, and the probability that the picture is identified as a souvenir picture may be output. In this case, when it is identified the probability that the preview picture is a souvenir picture exceeds a predetermined value, it is determined that the triggering condition is satisfied.

In addition, for example, if a voice recognition system of the image capturing apparatus 400 recognizes that the voice of the user when photographing includes an intention to photograph a souvenir picture, for example, the user says "photograph a souvenir picture for me", it is determined that the triggering condition is satisfied. Alternatively, the user says "the scenery here is beautiful", "we photographed here last year", "we should photograph here every year", etc., the voice recognition system may ask if the user needs to photograph a souvenir picture, if the user answers "Yes", then it is determined that the triggering condition is satisfied.

According to an embodiment of the present disclosure, the processor 470 may turn on or off at least one of the plurality of triggering conditions described above.

The processor 470 may provide a photographing prompt for the current photographing according to the associated picture.

According to an embodiment of the present disclosure, the processor may display a list of thumbnails including the associated picture on a preview picture of the current photographing, in response to the user selecting a picture from the list, and provide a photographing prompt based on the selected picture.

According to an embodiment of the present disclosure, the processor 470 may determine whether a template picture exists in the associated picture; in response to a template picture exists in the associated picture, arrange a thumbnail of the template picture in front of a thumbnail of a non-template picture in the associated picture in the list; in response to a template picture does not exist in the associated picture, determine the number of the associated picture; in response to the number of the associated picture does not exceed a predetermined value, display a list; and in response to the number of the associated picture exceeds the predetermined value, prompt the user to photograph a picture as a sample picture, match the sample picture with each of the associated picture after the user photographing the sample picture, and arrange the thumbnail of the associated picture in the list in descending order according to a matching similarity of the sample picture and each picture in the associated picture.

For example, if the associated picture includes a template picture, then the processor 470 may display the template picture in front of a non-template picture in the list, and the template picture may be marked with a corner mark, etc. to be distinguished from the non-template picture. If the associated picture includes a plurality of pictures of different backgrounds, in order to facilitate user selection, the processor 470 may discriminate pictures belonging to the same background by a feature point extraction and matching method (for example, ORB, SIFT, histogram, etc.), and display only one of the plurality of pictures of the same background. For example, if there is a template picture in the plurality of pictures of the same background, only the template picture is displayed. If there is no template picture in the plurality of pictures of the same background, a best one may be selected and displayed by judging the pros and cons of the pictures.

For example, if the number of associated picture exceeds 10, and the associated picture cannot be all displayed in the list on the preview picture, then the processor 470 may prompt the user to photograph a picture as the sample picture, and after the photographing, the processor 470 may start a process to perform "searching by picture" in the background of the image capturing apparatus 400. For example, first, a picture of a position near the position of the sample picture is searched in the associated picture, to obtain a set of pictures historically photographed near the current position, and then the feature point extraction method (for example, an algorithm such as ORB, SIFT, or histogram) is used to perform feature point extraction on the sample picture and all the pictures in the set, to obtain feature point descriptors. After the feature points are extracted, the feature point descriptor of the sample picture may be matched with the feature point descriptor of each picture in the set by a matching algorithm (for example, a k-proximity classification algorithm), the more the number of feature points successfully matched, the higher the similarity of the two pictures. The processor 470 may arrange the pictures in the set in descending order in the list based on the number of feature points that the sample picture successfully matches each picture in the set. However, there is a difference in the scenery in pictures photographed in different seasons, such as difference in color, or tree clipping condition. In this case, the processor 470 may discriminate the grayscale images by a feature point matching algorithm, thereby ignoring the difference in color. At the same time, in order to improve the matching efficiency, the processor 470 may match the thumbnails of the pictures to ignore a small amount of content change.

According to an embodiment of the present disclosure, after the user selects a picture from the list, the processor 470 may acquire a feature content of the selected picture, and overlapping display an outline of the feature content and/or text information about the feature content on the preview picture, so that the user may photograph a picture based on the outline or the text information overlapping displayed.

For example, the processor 470 may extract the outline by using an existing image processing method. For example, the processor 470 may first perform grayscale processing on the picture, and then extract edges using an image gradient algorithm. The processor 470 may overlapping display the background outline, the character movement outline, and the like of the selected picture in a watermark-like form on the preview picture, and/or may overlapping display a text in the preview picture to mark the position of each character. Therefore, the user may adjust the position and angle of the camera based on the background outline, adjust the movement of the character to be photographed based on the movement outline, adjust the stance of the character to be photographed based on the text, or the like, thereby photographing a picture having a high degree of coincidence with the selected picture.

According to an embodiment of the present disclosure, after the user photographs a picture according to the photographing prompt, the processor 470 may add labelling information to the photographed picture.

According to an embodiment of the present disclosure, the processor 470 may add the labelling information to the photographed picture based on a difference between photographing information on the selected picture and photographing information on the photographed picture. The photographing information may include at least one of: position information on where the picture was photographed, date information on when the picture was photographed, information on whether the date of the photographed picture belongs to a holiday and/or anniversary, scenery information in the photographed picture, character information in the photographed picture, activity content information in the photographed picture, remark information on the selected picture, or title information on the selected picture. The difference may be a difference in photographing time, a difference in photographing position, a difference in the number of photographed characters, a difference in a movement of photographed characters, or the like between the selected picture and the photographed picture. For example, if the user selects to provide a photographing prompt for a picture photographed 2 years ago stored in the image capturing apparatus 400 having a remark of souvenir picture of the first wedding anniversary, and photographs a new picture according to the photographing prompt, labelling information "souvenir picture of the third wedding anniversary" may be added to the newly photographed picture by the processor 470 based on the difference in the photographing time. According to an embodiment of the present disclosure, the labelling information may be title information or remark information on the picture, or explanation information added to the picture.

According to an embodiment of the present disclosure, the processor 470 may synthesize the photographed picture and the selected picture into a contrast picture, after the user photographs a picture according to the photographing prompt. Also, the processor 470 may add photographing information corresponding to the photographed picture and the selected picture to the synthesized contrast picture. For example, the processor 470 may synthesize a picture photographed 2 years ago having a remark of souvenir picture of the first wedding anniversary selected by the user and a souvenir picture of the third wedding anniversary photographed based on the souvenir picture of the first wedding anniversary into one picture. In the synthesized picture, on the left side is the souvenir picture of the first wedding anniversary, and on the right side is the souvenir picture of the third wedding anniversary. In addition, the year of photographing may be respectively added below the souvenir picture of the first wedding anniversary and the souvenir picture of the third wedding anniversary in the synthesized picture, for example, 2016 and 2018. However, the above examples are merely exemplary, and the present invention is not limited thereto. However, the above example is merely exemplary, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, after the user photographs a picture according to the photographing prompt, the processor 470 may prompt the user to mark the photographed picture as the template picture. If a template picture is included in the associated picture, the original template picture may be set as non-template picture in response to the user setting the photographed picture as the template picture. In addition, after the user photographs a picture according to the photographing prompt, a coincidence degree of the photographed picture and the selected picture may be displayed on the preview picture by the processor 470. For example, after a picture is photographed, the background outline, the character movement, or the like in the newly photographed picture may be extracted, the coincidence degree of the corresponding outline in the new and old pictures is calculated, and the calculated coincidence degree is displayed on the preview picture. After the user photographs a picture according to the photographing prompt, the processor 470 may also prompt the user to set a reminder for asking the user whether to photograph a souvenir picture every predetermined interval. For example, after the user has photographed a souvenir picture according to the photographing prompt, a prompt "Whether to set to photograph a souvenir picture in this background every other year?" may be displayed on the preview picture.

In addition, according to an embodiment of the present disclosure, if the processor 470 does not acquire the associated picture, a photographing prompt cannot be provided for the current photographing. In this case, the user is prompted to directly perform the current photographing and use a currently photographed picture as the template picture, or the user is prompted to directly perform the current photographing and select at least one picture from a plurality of pictures currently photographed as the template picture. In addition, it is also possible to add labelling information to the photographed template picture and/or set a reminder to prompt the user to ask the user whether to photograph a souvenir picture every predetermined interval.

According to the above image capturing apparatus, a photographing prompt may be provided based on a historically photographed picture, so that the user may photograph a picture having a high degree of coincidence with the historically photographed picture.

According to an embodiment of the present disclosure, a computer readable storage medium storing a computer program is also provided. The computer readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the auxiliary photographing method as described above. The computer readable storage medium is any data storage apparatus that can store data read by a computer system. Examples of the computer readable storage medium includes a read only memory, a random access memory, a read-only optical disk, a magnetic tape, a floppy disk, an optical data storage apparatus, and a carrier wave (such as data transmission over the Internet via a wired or wireless transmission path).

According to an embodiment of the present disclosure, a computer apparatus is also provided. The computing apparatus includes a processor and a memory for storing a computer program, the computer program, when executed by the processor, cause the processor to execute the computer program of the auxiliary photographing method as described above.

Although some exemplary embodiments of the present disclosure have been illustrated and described, those skilled in the art should understand that these embodiments may be modified without departing from the principles and spirit of the present disclosure defined by the claims and their equivalents.

Claims

An auxiliary photographing method for an image capturing apparatus, comprising:

acquiring scene information on a user's current photographing;

acquiring, based on the scene information, an associated picture associated with the scene information from historical pictures photographed by the user; and

providing an operation instruction based on an acquisition result of the associated picture.
The auxiliary photographing method according to claim 1, wherein the scene information comprises at least one of:

position information on where the current photographing is performed, picture information on the current photographing, date information on when the current photographing is performed, or voice information of the user at the time of photographing; and/or,

the associated picture is from at least one of:

a locally stored picture, a picture in a cloud album connected to the image capturing apparatus, or a picture stored in another apparatus associated with the image capturing apparatus;

providing, based on the associated picture, a corresponding photographing prompt to the user for the current photographing, if the associated picture exists; and/or,

prompting the user to directly perform the current photographing and using a currently photographed picture as a template picture, or prompting the user to directly perform the current photographing and select at least one picture from a plurality of pictures currently photographed as a template picture, if no associated picture exists.
The auxiliary photographing method according to claim 2, wherein, the date information comprises: information that a date of the current photographing is a holiday and/or anniversary, the picture information is a preview picture identified by the image capturing apparatus at the time of the current photographing, and the voice information of the user at the time of photographing comprises voice information that an intention of the current photographing is to photograph a souvenir　picture; and/or,

the associated picture comprises at least one of:

a picture, position information of which matches, within a certain range, the position information in the scene information where the current photographing is performed by the user;

a picture, a picture information of which matches the picture information in the scene information to a certain extent;

a picture, date information of which matches the date information in the scene information;

a picture, content information of which matches recognized voice information of the user at the time of photographing,

a picture identified by an artificial intelligence algorithm; and/or,

wherein, the acquiring, based on the scene information, an associated picture associated with the scene information from historical pictures photographed by the user comprises:

acquiring the associated picture in response to the scene information satisfying at least one of following plurality of triggering conditions: there exists a historical picture photographed within a predetermined range of a current photographing position, a current photographing position is within a predetermined range of a famous attraction marked in a server connected to the image capturing apparatus, a date of the current photographing belongs to a holiday and/or anniversary, there exists a historically photographed picture comprising a currently photographed scenery, a currently photographed character is marked as a member of a souvenir　picture, a number of members marked as members of a souvenir picture in currently photographed characters exceeds a predetermined number, a probability that a preview picture identified by the image capturing apparatus being a souvenir　picture exceeds a predetermined value, or the voice information on the user at the time of photographing comprises an intention to photograph a souvenir　picture; and/or,

wherein, the providing, based on the associated picture, a corresponding photographing prompt to the user for the current photographing comprise: displaying a list of thumbnails comprising the associated picture on a preview picture of the current photographing; and providing the photographing prompt based on the selected picture in response to the user selecting a picture from the list;

adding labelling information to the photographed picture, after the user photographs a picture according to the photographing prompt;

prompting the user to mark the photographed picture as a template picture, after the user photographs a picture according to the photographing prompt;

displaying a coincidence degree of the photographed picture and the selected picture on a preview picture, after the user photographs a picture according to the photographing prompt;

prompting the user to set a reminder for asking the user whether to photograph a souvenir picture every predetermined interval, after the user photographs a picture according to the photographing prompt;

synthesizing the photographed picture and the selected picture into a contrast picture, after the user photographs a picture according to the photographing prompt; and/or,

if there exists no associated picture, adding labelling information to the photographed template picture and/or setting a reminder to prompt the user whether to photograph a souvenir picture every predetermined interval.
The auxiliary photographing method according to claim 3, wherein operations of acquiring the associated picture in response to the scene information satisfying the at least one triggering condition comprise:

displaying a souvenir picture icon on the preview picture of the current photographing, in response to the scene information satisfying the at least one triggering condition; and acquiring the associated picture, in response to the user clicking the souvenir picture icon;

an operation of identifying a probability that a preview picture of the image capturing apparatus is a souvenir　picture comprises: identifying, by a statistical method or an artificial intelligence algorithm, the probability that the preview picture is the souvenir　picture; and/or,

turning on or off at least one of the plurality of triggering conditions;

the displaying a list of thumbnails comprising the associated picture on a preview picture of the current photographing comprise:

if a template picture exists in the associated picture, arranging a thumbnail of the template picture in front of a thumbnail of a non-template picture in the associated picture in the list; otherwise,

determining a number of the associated picture;

displaying a list of thumbnails comprising all of the associated picture if the number of the associated picture does not exceed a predetermined value; and

prompting the user to photograph a picture as a sample picture if the number of the associated picture exceeds the predetermined value, and matching the photographed sample picture with each of the associated picture and preferentially displaying a thumbnail of the associated picture having a high matching degree; and/or,

the providing the photographing prompt based on the selected picture, in response to the user selecting a picture from the list comprises:

acquiring a feature content of the selected picture, and displaying an outline of the feature content and/or text information about the feature content on the preview picture as overlapped; and/or,

acquiring a predetermined interval from photographing information.
The auxiliary photographing method according to claim 4, wherein the famous attraction comprises at least one of: a location where historical pictures are photographed and the number of the photographed pictures marked by the user as souvenir pictures reaches a certain threshold, a location marked by a database by default, or a popular tourist attraction; and/or,

wherein, the displaying a souvenir picture icon on the preview picture comprises: displaying indication information for prompting the user to enter a souvenir picture mode around the souvenir picture icon; and/or,

wherein, the adding labelling information to the photographed picture comprises: adding the labelling information to the photographed picture based on a difference between photographing information on the selected picture and photographing information on the photographed picture; and/or,

if a template picture is comprised in the associated picture, setting the original template picture as non-template picture in response to the user selecting to mark the photographed picture as the template picture; and/or,

wherein, the identifying, by an artificial intelligence algorithm, the probability that the preview picture is the souvenir　picture comprises: using a neural network model to identify the probability that the preview picture is the souvenir　picture; and/or,

wherein, the synthesizing the photographed picture and the selected picture into a contrast picture comprises: adding photographing information corresponding to the photographed picture and the selected picture, to the synthesized contrast picture.
The auxiliary photographing method according to claim 5, wherein the photographing information comprises at least one of: position information on where the picture is photographed, date information on when the picture is photographed, information on whether a date of the photographed picture belongs to a holiday and/or anniversary, scenery information in the photographed picture, character information in the photographed picture, activity content information in the photographed picture, remark information on the selected picture, or title information on the selected picture;

wherein, the difference comprises at least one of: a difference in photographing time, a difference in photographing position, a difference in a number and/or movement of photographed characters, or a difference in a photographed scenery; and/or,

wherein, the neural network model is obtained by iteratively training the model with a set of pictures marked as souvenir pictures and a set of pictures not marked as souvenir pictures.
An image capturing apparatus, the apparatus comprising:

a display;

a processor, configured to:

acquire scene information on a user's current photographing;

acquire, based on the scene information, an associated picture associated with the scene information from historical pictures photographed by the user; and

provide an operation instruction based on an acquisition result of the associated picture.
The image capturing apparatus according to claim 7, wherein the scene information comprises at least one of position information, picture information, date information, or voice information of the user at the time of the current photographing;

the associated picture is from at least one of: a locally stored picture, a picture in a cloud album connected to the image capturing apparatus, or a picture stored in another apparatus associated with the image capturing apparatus;

wherein, the processor is further configured to:

provide a corresponding photographing prompt to the user for the current photographing based on the associated picture, if the associated picture exists; and/or,

prompt the user to directly perform the current photographing and use a currently photographed picture as a template picture, or prompt the user to directly perform the current photographing and select at least one picture from a plurality of pictures currently photographed as a template picture, if no associated picture exists.
The image capturing apparatus according to claim 8, wherein,

the date information comprises information that a date of the current photographing is a holiday and/or anniversary,

the picture information is a preview picture identified by the image capturing apparatus at the current photographing, and

the voice information of the user at the time of photographing comprises voice information that an intention of the current photographing is to photograph a souvenir　picture; and/or,

the associated picture comprises at least one of:

a picture, position information of which matches, within a certain range, the position information in the scene information, where the current photographing is performed by the user;

a picture, a picture information of which matches the picture information in the scene information to a certain extent;

a picture, date information of which matches the date information in the scene information;

a picture, content information of which matches recognized voice information of the user at the time of photographing, or

a picture identified by an artificial intelligence algorithm.
The image capturing apparatus according to claim 8, wherein, the processor is further configured to: acquire the associated picture in response to the scene information satisfying at least one of following plurality of triggering conditions: there exists a historical picture photographed within a predetermined range of a current photographing position, a current photographing position is within a predetermined range of a famous attraction marked in a server connected to the image capturing apparatus, a date of the current photographing belongs to a holiday and/or anniversary, there exists a historically photographed picture comprising a currently photographed scenery, a currently photographed character is marked as a member of a souvenir　picture, a number of members marked as members of a souvenir picture in currently photographed characters exceeds a predetermined number, a probability that a preview picture identified by the image capturing apparatus being a souvenir　picture exceeds a predetermined value, or the voice information of the user at the time of photographing comprises an intention to photograph a souvenir　picture.
The image capturing apparatus according to claim 8, wherein, the processor is further configured to:

display a list of thumbnails comprising the associated picture on a preview picture of the current photographing; and provide the photographing prompt based on the selected picture, in response to the user selecting a picture from the list;

add labelling information to the photographed picture, after the user photographs a picture according to the photographing prompt;

prompt the user to mark the photographed picture as a template picture, after the user photographs a picture according to the photographing prompt;

display a coincidence degree of the photographed picture and the selected picture on a preview picture, after the user photographs a picture according to the photographing prompt;

prompt the user to set a reminder to the user whether to photograph a souvenir picture every predetermined interval, after the user photographs a picture according to the photographing prompt;

synthesize the photographed picture and the selected picture into a contrast picture, after the user photographing a picture according to the photographing prompt; and/or

if no associated picture exists, add labelling information to the photographed template picture and/or set a reminder to prompt the user whether to photograph a souvenir picture every predetermined interval.
The image capturing apparatus according to claim 11, wherein the processor is further configured to:

display a souvenir picture icon on the preview picture of the current photographing, in response to the scene information satisfying the at least one triggering condition; and acquire the associated picture, in response to the user clicking the souvenir picture icon;

identify, by a statistical method or an artificial intelligence algorithm, the probability that the preview picture is the souvenir　picture; and/or,

turn on or off at least one of the plurality of triggering conditions.
The image capturing apparatus according to claim 9, wherein, the processor is further configured to:

if a template picture exists in the associated picture, arrange a thumbnail of the template picture in front of a thumbnail of a non-template picture in the associated picture in the list; and if a template picture does not exist in the associated picture, determine a number of the associated picture, and display a list of thumbnails comprising all of the associated picture if the number of the associated picture does not exceed a predetermined value, and prompt the user to photograph a picture as a sample picture if the number of the associated picture exceeds the predetermined value, and match the photographed sample picture with each of the associated picture, and preferentially display a thumbnail of the associated picture having a high matching degree;

acquire a feature content of the selected picture, and display an outline of the feature content and/or text information about the feature content on the preview picture as overlapped; and/or,

acquire the predetermined interval by photographing information.
The image capturing apparatus according to claim 13, wherein the famous attraction comprises: a location where historical pictures are photographed and the number of the photographed pictures marked by the user as souvenir pictures reaches a certain threshold, a location marked by a database by default, or a popular tourist attraction; and/or,

wherein, the processor is further configured to:

display indication information for prompting the user to enter a souvenir picture mode around the souvenir picture icon;

add the labelling information to the photographed picture based on a difference between photographing information on the selected picture and photographing information on the photographed picture;

if a template picture is comprised in the associated picture, set the original template picture as non-template picture in response to the user selecting to mark the photographed picture as the template picture;

use a neural network model to identify the probability that the preview picture is the souvenir　picture; and/or,

add photographing information corresponding to the photographed picture and the selected picture, to the synthesized contrast picture.
The image capturing apparatus according to claim 14, wherein the photographing information comprises at least one of: position information on the photographed picture, date information on the photographed picture, information on whether a date of the photographed picture belongs to a holiday and/or anniversary, scenery information in the photographed picture, character information in the photographed picture, activity content information in the photographed picture, remark information on the selected picture, or title information on the selected picture;

wherein, the difference comprises at least one of: a difference in photographing time, a difference in photographing position, a difference in a number and/or movement of photographed characters, or a difference in a photographed scenery; and/or,

wherein, the neural network model is obtained by iterative training the model with a set of pictures marked as souvenir pictures and a set of pictures not marked as souvenir pictures.