KR20230008687A - Method and apparatus for automatic picture labeling and recording in smartphone - Google Patents

Abstract

An apparatus and method for automatically labeling and recording a picture in a smartphone are disclosed. An apparatus for automatically labeling and recording a picture in a smartphone may include: a camera lens for taking a picture; a shooting button for instructing photo shoot; a display part for displaying a taken photo; a voice input button for instructing voice input and text conversion; a voice recognition part for converting the input voice into text or text string; and a file storage part for storing the converted character or character string as a file name in a photo file taken under the control of a control part.

Description

Photo automatic labeling and recording device and method on a smartphone {METHOD AND APPARATUS FOR AUTOMATIC PICTURE LABELING AND RECORDING IN SMARTPHONE}

The following embodiments relate to a technology for automatically naming or recording sound to pictures taken on a smartphone.

I take a lot of pictures in my daily life with my smartphone or digital camera. The subject of a photograph can be a person or an object or event. In general, when a picture is taken with a smartphone or camera, the date and time the picture was taken is saved as a picture file name or a picture serial number is saved as a file name by the built-in clock. Also, in some cameras or smart phones, the shooting time, camera property information, and shooting location information are stored in the form of metadata included in the photo.

1. Simultaneous photo/voice editing method on a mobile phone equipped with a camera, registration number 10-0341987, registration date June 12, 2002 2. Digital image capture session and related metadata, application number 10-2020-7006887, application date September 10, 2018 Patent Document 1 relates to a technique for simultaneously outputting audio and pictures when playing a picture by storing a picture file and a sound file together. In addition, Patent Document 2 recognizes meta data such as the name of an object from an image of an object input to a smartphone camera, connects to a server through a communication network, and receives and displays information such as price. It relates to a technology for additionally recognizing meta information such as the name of an object by converting voice into text.

However, the most necessary function for a smartphone camera user is to save a phrase or information about a corresponding photo in a photo file in the form of a photo name or meta data by which the user can distinguish between several different photos. Therefore, later, it is a very important function to classify photos by looking at the name or meta information of the photo file, to grasp information about the photo, or to search for one or more photos with a desired keyword among multiple photo files.

To this end, in current cameras or smart phones, once a photo is taken and a photo file is stored in the device, the user must use a file manager or photo playback function to modify the existing file name using a text editing function. In the case of a camera or smartphone, it is common that the text editing function is inconvenient due to the small size of the device. Also, it is very inconvenient to modify the photo file name using the text editing function every time a photo is taken. In addition, if you try to modify the file name at once after taking several pictures, it is difficult to remember and distinguish the exact contents of each picture.

In order to solve this inconvenience, as in Patent Document 1, in some cases, an audio file is stored together with a photo file, but it is very inconvenient to play the audio file individually to check the background information on the photo, which is very inconvenient. There is a problem in that the function cannot be used when the connection of the voice file is changed or the corresponding voice file is deleted. In addition, in the case of Patent Document 2, the use and method are completely different for the purpose of recognizing, searching, and purchasing objects included in the photo, not for distinguishing photo files.

Recently, voice recognition technology has been developed and accuracy has improved. After taking a picture with a camera or smartphone, say the name of the picture with the user's voice, and recognize it by the camera itself or by using a remote voice recognition server. If converted to text and saved as the name of a photo file or metadata of a photo, all of the above problems are solved. In addition, a voice or sound file corresponding to a photo or video is additionally created and integrated with the photo or video, so that the situation at the shooting site can be expressed more vividly when the photo or video is played.

An apparatus for automatically labeling and recording photos in a smartphone according to an embodiment includes a camera lens for taking photos; a photographing button instructing photographing; a display unit for displaying a photograph taken; a voice input button instructing voice input and character conversion; a voice recognition unit that converts input voice into text or text; and a file storage unit for storing the converted character or character string as a file name in a photograph file taken under the control of the controller.

A record button for recording a sound signal related to a photograph taken by a user's intention may be further included.

The user presses the voice input button at the same time as, before or after taking a photo, and says the name of the photo. Automatically saved as file name or meta data

may further include.

Clicking the voice input button starts recognizing the character string of the voice signal, clicking the corresponding button again ends voice recognition, or clicking the voice input button starts recognizing the voice and after a predetermined time has elapsed, the voice signal is no longer Ability to automatically end voice recognition when there is no input

may further include.

A function of generating a file name by performing string recognition through voice recognition for a predetermined time after taking a photo or video may be further included.

Voice recognition is performed by the voice recognition unit of the smartphone itself, or voice or sound signal data is transmitted to a conversion server in a remote place through the communication network of the smartphone under the leadership of the voice recognition unit, and the conversion server converts it into a character or string. Thereafter, a function to be received by the smartphone again may be further included.

When a plurality of features (a specific object, place, person, or event (act or event)) are extracted from a photographed photograph, the name of each feature is saved as the name of a plurality of photo files or stored in a single photo file. A function of storing a plurality of meta data may be further included.

Depending on the names or metadata given to the photo files taken in the above way, the photo files may be automatically classified and stored in different folders, or a serial number may be added before or after the name according to the time taken. . Alternatively, if a photo with the same file name already exists, a function of distinguishing the file by adding a serial number according to the shooting time or information on the shooting date may be further included at the end of the file name.

The sound signal input to the smartphone can be recorded using the above record button and saved as a separate sound file or included as meta data in a photo file. In the case of saving as a separate sound file, the corresponding photo or video file A sound file is stored with the same file name or meta data, and in the case of a photo, a function of generating a single file by including a recorded sound signal in a video format may be further included.

When the sound file is included in the photo or video file or the corresponding sound file exists, a function of including the existence or sound file name in metadata of the photo or video file may be further included.

Thumbnails can be displayed in the list of taken photos or videos. If you click the thumbnail or file name of a specific photo, the photo is played on the screen and the recorded sound file is played simultaneously. In the case of a specific video, the thumbnail or file name is displayed. When clicked, a function of playing the first screen of the video as a still image and simultaneously playing the sound file and then playing the rest of the video may be further included.

When a photo file containing a pre-determined keyword in the name or metadata of a photo file is created on a smartphone owned by one of several pre-determined user groups, it is automatically shared with the smartphones owned by members of the group. A synchronization function may be further included.

When a photo file containing a pre-determined keyword in the name or metadata of a photo file is created on a smartphone owned by one of a group of pre-determined users, it is automatically shared on the smartphone owned by the specific person. and may further include a function of being synchronized.

A method for automatically labeling and recording photos in a smartphone, comprising: taking a photo or video with a camera of the smartphone; Press the voice input button of the smartphone or input the voice into the smartphone for a certain period of time after taking the above picture, recognize it on the smartphone or an external server and convert it into a character string, and then the file name or metadata of the photo or video above Saving as; If the file name of the generated string is not correct or a change is desired, correcting the file name by inputting voice again; Optionally recording a voice or sound signal using a record button and storing it as a separate sound file with the same file name as the photo file name; When a thumbnail or the above file name is selected from a list of taken pictures or videos, the sound file recorded together with the picture is played.

According to an embodiment, voice information may be recognized as text information when a photo or video is taken and stored as a corresponding file name or metadata.

According to an embodiment, the name or meta data of the stored file is displayed so that the user can distinguish the file or grasp the information of the file.

According to an embodiment, files matching a specific keyword may be searched for or classified using the names of the stored files or meta data.

According to an embodiment, a sound signal may be additionally recorded and stored when a picture or video is taken so that the sound signal can be heard when the picture or video is reproduced.

1 is a diagram illustrating a configuration of a photo automatic labeling and recording device in a smart phone according to an embodiment.
2 is diagrams configuring a screen of a smart phone in an apparatus for automatically labeling and recording photos in a smart phone according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes can be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents or substitutes to the embodiments are included within the scope of rights.

Terms used in the examples are used only for descriptive purposes and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to indicate that there is a characteristic point, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of feature points, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

In addition, in the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term. When an element is described as being “connected,” “coupled to,” or “connected” to another element, that element may be directly connected or connected to the other element, but there may be another element between the elements. It should be understood that may be "connected", "coupled" or "connected".

Components included in one embodiment and components having common functions will be described using the same names in other embodiments. Unless stated to the contrary, descriptions described in one embodiment may be applied to other embodiments, and detailed descriptions will be omitted to the extent of overlap.

1 is a diagram illustrating a configuration of a photo automatic labeling and recording device in a smart phone according to an embodiment. A smartphone user presses the shooting button, uses a remote control, gives a signal with voice or sound, gives a signal with a camera input gesture, gives a signal with a smartphone movement, or sends a control command from a remote place through a communication method. Take a picture toward the object using the built-in camera. Photos are either still images or moving images. The user presses the voice input button at the same time as, before or after taking the photo and says the name of the photo, and the voice signal input by the microphone is processed by the voice recognition unit and converted into text or character string. The converted text or string is automatically saved as the name or meta data of the photographed picture file. Meta data refers to information other than image data included in a photo file. At this time, the file name or meta data may be deleted or modified through a user's confirmation step. In the case of a video file, it is processed in the same way.

If the name of the photo file created by voice recognition is automatically included in the metadata, even if the file name of the photo or video is changed later, the character string or file name originally input by voice recognition can be confirmed or restored.

Alternatively, when a voice input button is pressed, a sound signal input from the surroundings may be converted into a character or string and stored as a name or metadata of a photo file.

Alternatively, clicking the voice input button starts recognizing the text of the voice signal, and clicking the corresponding button again ends voice recognition, or clicking the voice input button starts voice recognition and after a predetermined time has elapsed, the voice signal no longer exists. If there is no input, voice recognition may be automatically terminated.

Alternatively, after taking a picture or video, the smartphone automatically generates a specific sound signal to start voice recognition, and the voice recognition may be stopped after a predetermined time or by clicking the voice input button by the user.

Instead of clicking the voice input button, use the remote control, give a signal to the smartphone with a specific word, voice or sound, give a signal with a specific body movement with the smartphone camera input, give a signal with a specific movement pattern of the smartphone, A voice signal input into a microphone by pressing a set button of a smartphone or by a control command transmitted through a communication method from a remote place may be processed by a voice recognition unit and converted into text.

While voice recognition is in progress, the color of the voice input button may be changed so that the user can recognize that voice is being recognized.

In addition, even if the user does not intentionally specify a voice recognition command to the smartphone, as a basic function, it recognizes the voice signal input to the smartphone during a certain period of time while taking a picture with the smartphone or while a voice above a certain level is recognized. It can be converted into text or string and saved as the name of a photo file or meta data.

After setting the file name or meta data by voice recognition after taking a picture or video, the user presses the voice input button again to create a string again by voice recognition in the same way as above, check the string file name displayed on the display, and confirm the previous file name. or metadata can be modified.

The main functions of taking pictures, converting text, and saving photo files may be performed in the main body of a smartphone or camera, or may be performed by a separate software application program stored in the main body, and the main functions are the main body and the app. It can also be distributed and executed in a program.

Alternatively, the voice or sound signal data may be transmitted to a conversion server in a remote location through a communication network of the smartphone, converted into text or character strings in the conversion server, and then received by the smartphone again.

In addition, the photo taking function may be performed by a remote smartphone camera, voice or sound signal input may be performed locally, and text conversion and photo file storage may be performed by a local conversion server.

Recently, many photos of various types are being collected in cloud servers, and if semi-supervised learning technology is applied, when a photo with characteristics of a specific object, place, person, or event (action or event) is input, it is previously stored It can automatically extract its features by comparing it to similar photos.

Therefore, the feature (specific object, place, person, or event (action or event)) of a photo taken on a smartphone is recognized by the smartphone or a remote conversion server, and the name of the feature is extracted, and the name or meta of the photo file is recognized. can be stored as data. Of course, even in this step, a correction step through user confirmation or voice recognition may be provided.

When a plurality of features (a specific object, place, person, or event (act or event)) are extracted from a photographed photograph, the name of each feature is saved as the name of a plurality of photo files or stored in a single photo file. A plurality of features may be stored as meta data.

Since the voice recognition rate may be lowered when the user's voice is uncertain or due to ambient noise, the accuracy of the file name intended by the user can be further increased by processing the above feature name extraction technology and voice recognition technology together.

Depending on the names or metadata given to the photo files taken in the above way, the photo files may be automatically classified and stored in different folders, or a serial number may be added before or after the name according to the time taken. . Alternatively, if a photo with the same file name already exists, a serial number according to the shooting time or shooting date information may be additionally attached to the file name to be distinguished.

On the other hand, during or after taking a photo or video, the sound signal input to the smartphone can be recorded using the record button of the smartphone and saved as a separate sound file or included as metadata in the photo file. there is. In the case of saving as a separate sound file, the sound file is saved with the same file name or meta data as the corresponding photo or video file.

Alternatively, separate link information between a photo or video file and a corresponding sound file may be created.

Alternatively, in the case of a photographed picture, it may be converted into a video format and a recorded sound signal may be included to generate a single file.

In addition, a voice signal used when a user converts a file name into a character string through voice recognition may be stored as a sound file.

In addition, if the sound file is included in the photo or video file or the corresponding sound file exists, whether or not or the name of the sound file may be included in the metadata of the photo or video file.

When displaying a list of pictures and video files taken in the above manner, the presence or absence of the sound signal may be displayed.

In addition, the date and time of the photo or video file may be additionally displayed on the list by means of metadata of the photo or video file.

In addition, thumbnails can be displayed in the above list. If you click the thumbnail or file name of a specific photo, the photo is played on the screen and the recorded sound file is played simultaneously. In the case of a specific video, clicking the thumbnail or file name will play the first video. After the screen is reproduced as a still image and the sound file is reproduced at the same time, the rest of the video is reproduced.

When a photo file containing a pre-determined keyword in the name or metadata of a photo file is created on a smartphone owned by one of several pre-determined user groups, it is automatically shared with the smartphones owned by members of the group. can be synchronized.

Alternatively, if a photo file containing a pre-determined keyword in the name or metadata of a photo file is created on a smartphone owned by one of several pre-determined user groups, the smartphone owned by the specific person is automatically created. can be shared and synchronized.

2 is diagrams configuring a screen of a smart phone in an apparatus for automatically labeling and recording photos in a smart phone according to an embodiment.

(a) of FIG. 2 is an initial screen of a smartphone app according to an embodiment, and includes four buttons. When the photo capture button is pressed, the camera app is moved to take a photo. If you take a picture, it automatically switches to the screen of (b). If you are satisfied with the picture you have taken, press the □Save□ button, and if you want to take another picture, press the □Retake button. If you take a picture again, you will return to the previous camera app. When □Save□ is executed, the screen of (c) is converted and the current file name is set to date and time information by default. To change this file name, press the □Voice file name□ button and say the desired file name. This voice is recognized as a character string and the current file name is automatically changed. If the voice recognition file name is wrong or you want to change it to another name, press □Voice File Name□ again and speak. If you want to additionally record your voice or ambient sounds, press the □Voice Recording□ button. Recording is done while the button is pressed, or it starts recording when the first button is pressed and stops when the second button is pressed. (d) is a screen displaying a list of photographed pictures or moving picture files. Thumbnails are also displayed, and the file name, file format, presence or absence of the corresponding audio file, date and time, and location are displayed. Clicking on a thumbnail picture or listing will play the picture or video along with a recorded audio file. (e) is the setting screen. If □on□ is selected for □Auto Recognition Mode□, there is no need to press the □Voice File Name□ button to recognize the photo file name as voice. When the button is pressed, the voice signal input for 5 seconds is automatically converted into a character string and displayed on the □Current File Name□ on the screen (c). In the case of □off□, it operates in the basic mode described above. Also, if you select □on□ for □Photo/Sound Synthesis□, the recorded sound file is made into a single video file. (For example, mpeg) In the case of □off□, a separate mp3 file with the same file name as the photo is created.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (12)

In the method of recording a sound signal in a photo or video file,
Recording an input sound signal using a record button;
Storing it as a separate sound file or including it as meta data in a photo file;
In the case of storing as a separate sound file, saving the sound file with the same file name or meta data as the corresponding photo or video file; and
When displaying a list of pictures and video files taken in the above manner, displaying whether a sound signal is present or included
A method for recording sound signals in a photo or video file containing According to claim 1,
If the sound file is included in the photo or video file or the sound file exists, including the presence or sound file name in metadata of the photo or video file
A method of recording a sound signal in a photo or video file that further includes. According to claim 1,
Generating separate link information between the photo or video file and the corresponding sound file
A method of recording a sound signal in a photo or video file that further includes. According to claim 1,
In the case of a photo, converting it into a video format and including the recorded sound signal to create a single file
A method of recording a sound signal in a photo or video file that further includes. According to claim 1,
Thumbnails can be displayed in the list of taken photos or videos, and when a thumbnail or file name of a specific photo is clicked, the photo is played on the screen and the recorded sound file is played at the same time. In the case of a specific video, the thumbnail or When the file name is clicked, the first screen of the video is played as a still image, the sound file is played at the same time, and the rest of the video is played.
A method of recording a sound signal in a photo or video file that further includes. According to claim 5,
Additionally, displaying the date and time or location of the photo or video file in the list by means of metadata of the photo or video file.
A method of recording a sound signal in a photo or video file that further includes. According to claim 1,
When a photo file containing a pre-determined keyword in the name or metadata of a photo file is created on a smartphone owned by one of several pre-determined user groups, it is automatically shared with the smartphones owned by members of the group. Synchronized Steps
A method of recording a sound signal in a photo or video file that further includes. According to claim 1,
When a photo file containing a pre-determined keyword in the name or metadata of a photo file is created on a smartphone owned by one of a group of pre-determined users, it is automatically shared on the smartphone owned by the specific person. Steps to be synchronized
A method of recording a sound signal in a photo or video file that further includes. In the device for recording a sound signal in a photo or video file,
Record button for recording the sound signal input to the smartphone;
It can be saved as a separate sound file or included as meta data in the photo file. In case of saving as a separate sound file, the sound file is saved with the same file name or meta data as the photo or video file. a file storage unit capable of generating a single file by including a recorded sound signal converted into a video format; and
Display unit for displaying whether a sound signal is present or included when displaying a list of pictures and video files taken in the above manner
A device for recording sound signals in a photo or video file containing a The file storage unit of claim 9,
A function of including the presence or sound file name in the metadata of the photo or video file if the above sound file is included in the photo or video file or the corresponding sound file exists
A device for recording a sound signal in a photo or video file further comprising a. The file storage unit of claim 9,
In the case of photos, a function that can be converted into a video format and created as a single file by including recorded sound signals
A device for recording a sound signal in a photo or video file further comprising a. The display unit of claim 9,
Thumbnails can be displayed in the list of taken photos or videos, and when a thumbnail or file name of a specific photo is clicked, the photo is played on the screen and the recorded sound file is played at the same time. In the case of a specific video, the thumbnail or A function to play the first screen of a video as a still image when the file name is clicked, play the above sound file at the same time, and then play the rest of the video
A device for recording a sound signal in a photo or video file further comprising a.

Images