KR20230117877A - Electronic device for generating composed audio video (av) file in which audio and video are synchronized - Google Patents

Abstract

In various embodiments, an electronic device may include cameras; mike; a processor operatively connected to the cameras and the microphone; and a memory operatively coupled to the processor. The memory, when executed, causes the processor to: generate a first video file by encoding a video signal received from a first one of the cameras frame by frame, and encode an audio signal received from the microphone frame by frame A first audio file is generated, the first video file and the first audio file are combined into a first AV file, stored in the memory, and the first video file and the first audio file are generated at the same time. , A second video file is generated by encoding a video signal received from a second camera of the cameras in units of frames, and a second audio file is generated by encoding an audio signal received from the microphone in units of frames, 2 video files and the second audio file are combined into a second AV file and stored in the memory, and a first video timestamp indicating a time point at which a first frame was generated in the first video file, a first video timestamp in the first audio file A first audio timestamp indicating when a frame was created, a second video timestamp indicating when a first frame was created in the second video file, and a second audio timestamp indicating when a first frame was created in the second audio file. An audio timestamp is stored in the memory, and a relatively slow point in time among video timestamps included in AV files or among video timestamps and audio timestamps included in AV files stored in the memory at the same time period Determines an unsynchronized video frame period based on a first timestamp indicating a relatively slow among audio timestamps included in the AV files or among video timestamps and audio timestamps included in the AV files. An asynchronous audio frame interval is determined based on a second timestamp indicating a point of view, a plurality of video signals are generated by decoding video frames not belonging to the asynchronous video frame interval for each of the AV files, and the plurality of video signals are generated. synthesized into one video signal, generating a synthesized video file by encoding the synthesized video signal, and generating a plurality of audio signals by decoding audio frames that do not belong to an asynchronous audio frame interval for each AV file, , synthesizing the plurality of audio signals into one audio signal, generating a synthesized audio file by encoding the synthesized audio signal, combining the synthesized video file and the synthesized audio file into a synthesized AV file, and storing the synthesized audio signal in the memory. You can store instructions that cause it to be saved. Besides that, various embodiments are possible.

Description

Electronic device for generating a composite audio video (AV) file in which audio and video are synchronized

Various embodiments relate to an electronic device that synthesizes a plurality of audio video (hereinafter referred to as AV (audio video)) files into one AV file and reproduces the AV file.

An electronic device (eg, a smart phone) may include a plurality of cameras. For example, the electronic device may include a front camera disposed on the front side of the electronic device and one or more rear cameras disposed on the rear side of the electronic device.

An electronic device may include a plurality of recorders. For example, the first recorder generates a first video file by encoding a video signal received from a first camera (eg, a front camera) and generates a second video file synchronized to the first video file by encoding an audio signal received from a microphone. You can create audio files. The first recorder may combine the first video file and the first audio file into one first AV file and include (or store) the first AV file in a container (eg, MP4). The second recorder generates a second video file by encoding a video signal received from a second camera (eg, a rear camera) and generates a second audio file synchronized with the second video file by encoding an audio signal received from a microphone. can do. The second recorder may combine the second video file and the second audio file into the second AV file and include them in the container.

The electronic device may include an editor for synthesizing a plurality of AV files into one AV file. For example, the editor may obtain the first video signal and the first audio signal by decoding the first AV file, and obtain the second video signal and the second audio signal by decoding the second AV file. The editor may create a third video file by synthesizing the first video signal and the second video signal into one video signal and encoding the synthesized video signal. The editor may create a third audio file by synthesizing the first audio signal and the second audio signal into one audio signal and encoding the synthesized audio signal. The editor may combine the third video file and the third audio file into a third AV file and include it in the container. The electronic device may reproduce the third AV file obtained as an editing result through a speaker and a display.

An electronic device may receive a recording command from a user through an input device (eg, a touch screen). Even if a recording command is given to a plurality of cameras at the same time, a point in time at which a subject is started to be photographed may be different for each of the plurality of cameras. For example, the first camera may start recording in response to a recording command, generate a video signal in units of frames from the start of recording, and transmit the video signal to the first recorder. The second camera responds relatively slowly to the recording command and starts recording, and generates a video signal in units of frames from the start of recording and transmits the video signal to the second recorder. Accordingly, a video signal transferred from the second camera to the second recorder may not include as many scenes as the difference in reaction speed between the two recorders. Such a difference in response speed may cause audio and video to become out of synch (aka AV sync mismatch) when the third AV file, which is the editing result, is reproduced. As a result, the electronic device may give the user discomfort due to the AV sync mismatch while reproducing the third AV file. For example, an operation and sound of one of the subjects may be temporally out of sync. A motion of a subject photographed by the first camera and a motion of a subject photographed by the second camera may be temporally different.

An order in which cameras start shooting and a time difference between shooting start points may be obtained through several tests. When synthesizing AV files, the shooting start order and time difference obtained through several experiments may be considered. For example, an average delay value indicating that the first camera starts shooting before the second camera and a time difference between shooting points may be obtained. The editor may synthesize a second video file with the rest of the first video file generated by using the first camera except for the front part corresponding to the parallax. However, the shooting start sequence and/or the time difference between shooting start points may not be the same every time shooting. Accordingly, it may not be possible to guarantee that synchronization in the AV file, which is the result of synthesis, will be made every time it is synthesized.

In various embodiments, the electronic device can solve the user's inconvenience caused by the AV sync mismatch by generating a synthesized file in which audio and video are synchronized.

The technical problem to be achieved in the present disclosure is not limited to the above-mentioned technical problem, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

In various embodiments, an electronic device may include cameras; mike; a processor operatively connected to the cameras and the microphone; and a memory operatively coupled to the processor. The memory, when executed, causes the processor to: generate a first video file by encoding a video signal received from a first one of the cameras frame by frame, and encode an audio signal received from the microphone frame by frame A first audio file is generated, the first video file and the first audio file are combined into a first AV file, stored in the memory, and the first video file and the first audio file are generated at the same time. , A second video file is generated by encoding a video signal received from a second camera of the cameras in units of frames, and a second audio file is generated by encoding an audio signal received from the microphone in units of frames, 2 video files and the second audio file are combined into a second AV file and stored in the memory, and a first video timestamp indicating a time point at which a first frame was generated in the first video file, a first video timestamp in the first audio file A first audio timestamp indicating when a frame was created, a second video timestamp indicating when a first frame was created in the second video file, and a second audio timestamp indicating when a first frame was created in the second audio file. An audio timestamp is stored in the memory, and a relatively slow point in time among video timestamps included in AV files or among video timestamps and audio timestamps included in AV files stored in the memory at the same time period Determines an unsynchronized video frame period based on a first timestamp indicating a relatively slow among audio timestamps included in the AV files or among video timestamps and audio timestamps included in the AV files. An asynchronous audio frame interval is determined based on a second timestamp indicating a point of view, a plurality of video signals are generated by decoding video frames not belonging to the asynchronous video frame interval for each of the AV files, and the plurality of video signals are generated. synthesized into one video signal, generating a synthesized video file by encoding the synthesized video signal, and generating a plurality of audio signals by decoding audio frames that do not belong to an asynchronous audio frame interval for each AV file, , synthesizing the plurality of audio signals into one audio signal, generating a synthesized audio file by encoding the synthesized audio signal, combining the synthesized video file and the synthesized audio file into a synthesized AV file, and storing the synthesized audio signal in the memory. You can store instructions that cause it to be saved.

In various embodiments, a method for operating an electronic device includes generating a first video file by encoding a video signal received from a first camera among cameras included in the electronic device in units of frames, and generating an audio signal received from the microphone. generating a first audio file by encoding in units of frames, combining the first video file and the first audio file into a first AV file, and storing the combined first AV file in a memory of the electronic device; A second video file is generated by encoding a video signal received from a second camera among the cameras frame by frame during the same time period in which the first video file and the first audio file are generated, and the audio received from the microphone is generated. generating a second audio file by encoding a signal frame by frame, combining the second video file and the second audio file into a second AV file, and storing the second audio file in the memory; A first video timestamp indicating when the first frame was created in the first video file, a first audio timestamp indicating when the first frame was created in the first audio file, and the first frame in the second video file storing, in the memory, a second video timestamp indicating when a first frame was created and a second audio timestamp indicating when a first frame was generated in the second audio file; Based on a first timestamp indicating a relatively slow time point among video timestamps included in AV files stored in the memory at the same time, or among video timestamps and audio timestamps included in the AV files, the ratio Determines a synchronization video frame interval, based on a second timestamp indicating a relatively slow time point among audio timestamps included in the AV files or among video timestamps and audio timestamps included in the AV files determining an unsynchronized audio frame section by performing the step; For each of the AV files, a plurality of video signals are generated by decoding video frames that do not belong to an asynchronous video frame period, the plurality of video signals are synthesized into one video signal, and the video signal synthesized into one is synthesized by encoding. creating a video file; For each of the AV files, a plurality of audio signals are generated by decoding audio frames that do not belong to the asynchronous audio frame period, the plurality of audio signals are synthesized into one audio signal, and the synthesized audio signal is synthesized by encoding. creating an audio file; and combining the composite video file and the composite audio file into a composite AV file and storing the composite AV file in the memory.

Various embodiments of the present invention may provide an electronic device capable of solving user's inconvenience due to AV sync mismatch by generating a synthesized file in which audio and video are synchronized. In addition to this, various effects identified directly or indirectly through this document may be provided.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
2 is a block diagram of an audio module, in accordance with various implementations.
3 is a block diagram illustrating a camera module, in accordance with various embodiments.
4A is a perspective view of a front side of a portable electronic device having a bar-type housing structure.
4B is a perspective view of the back of the electronic device of FIG. 4A.
5 is a block diagram of an electronic device configured to generate an AV synchronized composite AV file, according to one embodiment.
FIG. 6 illustrates an operation in which the recording module of FIG. 5 creates original AV files marked with timestamps.
7 and 8 illustrate an operation of determining an asynchronous frame period and generating a synthesized AV file using the asynchronous frame period.
9 is a flowchart illustrating operations for determining an unsynchronized frame period, according to an exemplary embodiment.
10 is a flowchart illustrating operations for generating an AV sync-matched composite AV file, according to an embodiment.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, processor 120 transfers instructions or data received from other components (eg, sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to an embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of other functionally related components (eg, the camera module 180 or the communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, a software structure in addition to a hardware structure.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used for a component (eg, the processor 120) of the electronic device 101 from an outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display module 160 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more specified protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or movement) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 may be a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, a : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, a legacy communication module). It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 may be used to realize peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency (for realizing URLLC). Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a bottom surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 is a block diagram 200 of an audio module 170, in accordance with various implementations. Referring to FIG. 2 , the audio module 170 includes, for example, an audio input interface 210, an audio input mixer 220, an analog to digital converter (ADC) 230, an audio signal processor 240, and a DAC. (digital to analog converter) 250, an audio output mixer 260, and/or an audio output interface 270 may be included.

The audio input interface 210 is a part of the input module 150 or through a microphone configured separately from the electronic device 101 (eg, a dynamic microphone, a condenser microphone, or a piezo microphone), obtained from the outside of the electronic device 101. An audio signal corresponding to sound may be received. For example, when an audio signal is acquired from an external electronic device 102 (eg, a headset or a microphone), the audio input interface 210 connects the external electronic device 102 to a connection terminal (eg, as shown in FIG. 1 ). An audio signal may be received by being connected directly through the connection terminal 178 or wirelessly (eg, Bluetooth communication) through a wireless communication module (eg, the wireless communication module 192 of FIG. 1 ). According to one embodiment, the audio input interface 210 receives a control signal related to an audio signal obtained (or received) from the external electronic device 102 (eg, a volume control signal received through an input button). can do. The audio input interface 210 includes a plurality of audio input channels, and can receive different audio signals for each corresponding audio input channel among the plurality of audio input channels. According to an embodiment, additionally or alternatively, the audio input interface 210 may receive audio signals from other components (eg, the processor 120 or the memory 130 of FIG. 1 ) of the electronic device 101 . there is.

The audio input mixer 220 may synthesize a plurality of input audio signals into at least one audio signal. For example, according to one embodiment, the audio input mixer 220 may synthesize a plurality of analog audio signals input through the audio input interface 210 into at least one analog audio signal.

The ADC 230 may convert an analog audio signal into a digital audio signal. For example, according to one embodiment, ADC 230 converts an analog audio signal received via audio input interface 210 or an analog audio signal synthesized via audio input mixer 220 additionally or alternatively to digital audio. can be converted into signals.

The audio signal processor 240 may perform various processes on the digital audio signal received through the ADC 230 or the digital audio signal received from other components of the electronic device 101 . For example, according to one embodiment, the audio signal processor 240 changes the sampling rate of one or more digital audio signals, applies one or more filters, performs interpolation processing, amplifies or attenuates all or some frequency bands, It can perform noise processing (eg, noise or echo reduction), channel change (eg, switching between mono and stereo), mixing, or specified signal extraction. According to one embodiment, one or more functions of the audio signal processor 240 may be implemented in the form of an equalizer.

The DAC 250 may convert a digital audio signal into an analog audio signal. For example, according to one embodiment, the DAC 250 is a digital audio signal processed by the audio signal processor 240, or other components of the electronic device 101 (eg, the processor 120 or the memory 130). )) to convert the digital audio signal obtained from the analog audio signal.

The audio output mixer 260 may synthesize a plurality of audio signals to be output into at least one audio signal. For example, according to one embodiment, the audio output mixer 260 includes an audio signal converted to analog through the DAC 250 and another analog audio signal (eg, an analog audio signal received through the audio input interface 210). ) into at least one analog audio signal.

The audio output interface 270 transmits the analog audio signal converted through the DAC 250 or the analog audio signal synthesized by the audio output mixer 260 additionally or alternatively to the electronic device 101 through the sound output module 155. ) can be output to the outside. The sound output module 155 may include, for example, a speaker or receiver such as a dynamic driver or a balanced armature driver. According to one embodiment, the sound output module 155 may include a plurality of speakers. In this case, the audio output interface 270 may output an audio signal having a plurality of different channels (eg, stereo or 5.1 channels) through at least some of the plurality of speakers. According to one embodiment, the audio output interface 270 is directly connected to the external electronic device 102 (eg, an external speaker or headset) through a connection terminal 178 or wirelessly through a wireless communication module 192. and output an audio signal.

According to one embodiment, the audio module 170 does not separately include the audio input mixer 220 or the audio output mixer 260, and uses at least one function of the audio signal processor 240 to generate a plurality of digital audio signals. At least one digital audio signal may be generated by combining them.

According to one embodiment, the audio module 170 is an audio amplifier (not shown) capable of amplifying an analog audio signal input through the audio input interface 210 or an audio signal to be output through the audio output interface 270. (e.g. speaker amplification circuit). According to one embodiment, the audio amplifier may be configured as a separate module from the audio module 170.

3 is a block diagram 300 illustrating a camera module 180, in accordance with various embodiments. Referring to FIG. 3 , the camera module 180 includes a lens assembly 310, a flash 320, an image sensor 330, an image stabilizer 340, a memory 350 (eg, a buffer memory), or an image signal processor. (360). The lens assembly 310 may collect light emitted from a subject that is an image capturing target. Lens assembly 310 may include one or more lenses. According to one embodiment, the camera module 180 may include a plurality of lens assemblies 310 . In this case, the camera module 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 310 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may have the same lens properties as other lens assemblies. may have one or more lens properties different from the lens properties of . The lens assembly 310 may include, for example, a wide-angle lens or a telephoto lens.

The flash 320 may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash 320 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp. The image sensor 330 may obtain an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 310 into an electrical signal. According to one embodiment, the image sensor 330 is, for example, an image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, It may include a plurality of image sensors having attributes, or a plurality of image sensors having other attributes. Each image sensor included in the image sensor 330 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 340 moves at least one lens or image sensor 330 included in the lens assembly 310 in a specific direction in response to movement of the camera module 180 or the electronic device 101 including the same. Operation characteristics of the image sensor 330 may be controlled (eg, read-out timing is adjusted, etc.). This makes it possible to compensate at least part of the negative effect of the movement on the image being taken. According to an embodiment, the image stabilizer 340 uses a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180 to control the camera module 180 or the electronic device 101 . ) can be detected. According to one embodiment, the image stabilizer 340 may be implemented as, for example, an optical image stabilizer. The memory 350 may at least temporarily store at least a portion of an image acquired through the image sensor 330 for a next image processing task. For example, when image acquisition is delayed according to the shutter, or a plurality of images are acquired at high speed, the acquired original image (eg, a Bayer-patterned image or a high-resolution image) is stored in the memory 350 and , a copy image (eg, a low resolution image) corresponding thereto may be previewed through the display module 160 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a part of the original image stored in the memory 350 may be acquired and processed by, for example, the image signal processor 360 . According to one embodiment, the memory 350 may be configured as at least a part of the memory 130 or as a separate memory operated independently of the memory 130 .

The image signal processor 360 may perform one or more image processes on an image acquired through the image sensor 330 or an image stored in the memory 350 . The one or more image processes, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring, sharpening, or softening). Additionally or alternatively, the image signal processor 360 controls (eg, exposure time control or read-out timing control) for at least one of the components included in the camera module 180 (eg, the image sensor 330). etc.) can be performed. Images processed by the image signal processor 360 are stored again in the memory 350 for further processing or external components of the camera module 180 (eg, memory 130, display module 160, electronic device ( 102), the electronic device 104, or the server 108). According to one embodiment, the image signal processor 360 may be configured as at least a part of the processor 120 or as a separate processor operated independently of the processor 120 . When the image signal processor 360 is configured as a separate processor from the processor 120, at least one image processed by the image signal processor 360 is displayed by the processor 120 as it is or after additional image processing. It can be displayed via module 160 .

According to an embodiment, the electronic device 101 may include a plurality of camera modules 180 each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 180 may be a wide-angle camera, and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules 180 may be a front camera, and at least another one may be a rear camera.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numerals may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish that component from other corresponding components, and may refer to that component in other respects (eg, importance or order) is not limited. A (eg, first) component is said to be "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeably interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integral part or the smallest unit of a part or part thereof that performs one or more functions. According to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component among the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

Various housing structures may be applied to a portable electronic device (eg, the electronic device 101 of FIG. 1 ). For example, the portable electronic device may have a so-called bar-type housing structure, a foldable housing structure, or a slidable (or rollable) housing structure. In one embodiment, the bar-type housing structure includes a first cover forming a front surface of the portable electronic device, a second cover forming a rear surface of the portable electronic device, and a side bezel forming a side surface of the portable electronic device. structure may be included. In one embodiment, the foldable housing structure may include a first housing, a second housing, and a hinge assembly that makes the two housings rotatable. In the foldable housing structure, a first part of a display (eg, a flexible display) may be disposed in a first housing, and a second part of the display may be disposed in a second housing. The foldable housing structure may be implemented in an in-folding method in which a first part and a second part face each other when the portable electronic device is folded. Alternatively, the foldable housing structure may be implemented in an out-folding manner in which the first part and the second part face each other in opposite directions when the portable electronic device is folded. In one embodiment, the slideable housing structure may include a housing, a slider portion, and a roller through which a portion of the slider portion is drawn into or drawn out of the housing. As the slider is drawn into the housing, a portion of the display (eg, the flexible display) may enter the housing.

Hereinafter, for convenience of description, a surface on which a display (eg, a flexible display) is visually exposed to a user may be referred to as a front surface (or first surface) of the portable electronic device. Also, a surface opposite to the front surface may be referred to as a rear surface (or a second surface) of the portable electronic device. Also, a surface surrounding a space between the front and rear surfaces may be referred to as a side surface of the portable electronic device. In one embodiment, the portable electronic device may further include a separate sub-display visually exposed through the rear surface.

4A is a perspective view of the front of a portable electronic device 400 having a bar-type housing structure. 4B is a perspective view of the back of the electronic device 400 of FIG. 4A.

Referring to FIGS. 4A and 4B , a housing 410 of a portable electronic device 400 (eg, the electronic device 101 of FIG. 1 ) has a first surface (or front surface) 410A, and a second surface (or electronic device 101). It may include a rear surface) 410B, and a side surface 410C surrounding a space between the first surface 410A and the second surface 410B. According to one embodiment, the first surface 410A may be composed of a front plate (eg, a glass plate or a polymer plate including various coating layers) that is substantially transparent at least in part. The second face 410B may be composed of a substantially opaque back plate. The side surface 410C is combined with the front plate and the back plate and may be composed of a side bezel structure (or “side member”) including metal and/or polymer.

According to an embodiment, the portable electronic device 400 includes a display 401, a microphone hole 403, speaker holes 407 and 414, sensor modules 404 and 419, and camera modules 405, 412 and 413. , a key input device 417 , and a connector 408 . In some embodiments, the portable electronic device 400 may omit at least one of the components (eg, the key input device 417) or may additionally include other components.

The display 401 (eg, the display module 160 of FIG. 1 ) may be exposed through the first surface 410A. The display 401 may be combined with or disposed adjacent to a touch sensing circuit, a pressure sensor capable of measuring the intensity (pressure) of a touch, and/or a digitizer that detects a magnetic field type stylus pen.

The sensor modules 404 and 419 (eg, the sensor module 176 of FIG. 1 ) may generate electrical signals or data in response to an internal operating state of the portable electronic device 400 or an external environmental state. . In one embodiment, the sensor modules 404 and 419 are coupled to a first sensor module 404 (eg, a proximity sensor and/or a fingerprint sensor) disposed on the first side 410A and/or on the second side 410B. A disposed second sensor module 419 (eg, an HRM sensor) may be included. The first sensor module 404 may be disposed under the display 401 when viewing the display 401 from the first surface 410A.

The camera modules 405, 412, and 413 (eg, the camera module 180 of FIG. 1) include a front camera 405 disposed on the first surface 410A and one or more rear cameras disposed on the second surface 410B. (412) and flash (413). The cameras 405 and 412 may include a lens assembly (including one or a plurality of lenses), an image sensor, and/or an image signal processor. The flash 413 may include, for example, a light emitting diode or a xenon lamp. In some embodiments, two or more lenses (eg, a wide-angle lens, an ultra-wide-angle lens, or a telephoto lens) and image sensors may be disposed on one side of the electronic device 400 . In one embodiment, the front camera 405, when viewing the display 401 from above the first side 410A, is positioned under the display 401 to receive light through the display 401 (UDC). display camera). A hole (eg, a punch hole) (or opening) may be formed in a portion of the display 401 facing the front camera 405 . For example, the display 401 includes several layers (e.g., a polarizing film, a display panel, a subsidiary material layer (e.g., a light blocking layer for blocking light generated from the display panel or light incident on the display panel from the outside), and a heat dissipation sheet. , Sponge)), and in the display 401, through-holes may be formed in the remaining layers except for at least one layer (eg, the display panel). As another example, through holes (eg, punch holes) may be formed in all layers. At least a part (eg, lens) of the front camera 405 may be disposed on an inner space of a hole drilled in the display 401 . Although not shown, in some embodiments, a plurality of front cameras may be disposed below the display 401 . An electronic device (eg, the electronic device 101 of FIG. 1 ) may include a rear display. Accordingly, a UDC that receives light through the rear display may be additionally disposed under the rear display. The rear camera 412 may include an array camera having a plurality of lenses arranged on the rear surface (eg, arranged side by side in a straight line as shown).

Although not shown, a front camera and/or a rear camera may be configured in a portable electronic device having a foldable housing structure or a slideable housing structure.

5 is a block diagram of an electronic device 500 configured to generate an AV synchronized composite AV file, according to one embodiment. FIG. 6 illustrates an operation in which the recording module 503 of FIG. 5 creates original AV files 630 and 640 marked with timestamps. In various embodiments of the present document, the timestamp may include an audio timestamp indicating when the first audio frame was created in the original AV file and a video timestamp indicating when the first video frame was created. In various embodiments of the present document, the first frame may be defined as a frame generated first in time in the corresponding original AV file. For example, the first audio frame may be an audio frame first generated when the corresponding recorder starts recording. The first video frame may be a video frame first created when the corresponding recorder starts recording. 7 and 8 show asynchronous frames that may adversely affect the synchronization of audio and video when the asynchronous section determination module 506 of FIG. 5 synthesizes original AV files 630 and 640 using timestamps. The operation of determining the interval is exemplified. Also, FIGS. 7 and 8 illustrate an operation of the editing module 505 of FIG. 5 generating one first composite AV file 750 by using the remaining frame sections except for the unsynchronized frame section. In addition, FIGS. 7 and 8 illustrate an operation of generating a second composite AV file 760 that further includes an asynchronous frame section and maintains synchronization between audio and video.

Referring to FIG. 5 , the electronic device 500 includes a camera module 501, an audio module 502, a recording module 503, a time provision module 504, an editing module 505, and an asynchronous section determination module 506. ), a playback module 507, a display 577, a memory 588, and a processor 599. In various embodiments, the components of the electronic device 500 for generating a synthesized AV file in which audio and video are synchronized may be operatively or electrically connected to each other. According to one embodiment, modules 503 - 507 may be program modules executed on processor 599 (eg, processor 120 of FIG. 1 ). For example, at least one of the modules 503 to 507 is stored as instructions in a memory 588 (eg, the memory 130 of FIG. 1), and a processor 599 (eg, the processor of FIG. 1 ( 120)).

The camera module 501 (eg, the camera module 180 of FIG. 1 ) may include a plurality of cameras (eg, the cameras 405 and 412 of FIG. 4 ).

The audio module 502 (eg, the audio module 170 of FIG. 1) receives a microphone (or an external device (eg, a wireless headset) through a wireless communication circuit (eg, the wireless communication module 192 of FIG. 1)). The received analog audio signal may be converted into a frame-unit digital audio signal and output to another component (eg, the recording module 503). The audio module 502 converts a digital audio signal received from another component (eg, the processor 599) into an analog audio signal, and converts a speaker (or a wireless communication circuit (eg, the wireless communication module 192 of FIG. 1)) into an analog audio signal. can be output to an external device).

The recording module 503 may include a plurality of recorders. Recorders may be provided in the recording module 503 as many as the number of cameras and may be assigned (eg, operatively connected) to each of the cameras. The recorder encodes the video signal in units of frames received from the camera designated for itself using the designated video codec (eg, the compression format of the video file included (or stored) in the container 590 (eg, H.264, AV1) ) to generate a video file. In addition, the recorder encodes the frame-unit audio signal received from the microphone 510 through the audio module 502 using a designated audio codec (eg, a compression format of an audio file included in the container 590 (eg, AAC) , MP3 encoding) to create an audio file. The recorder may combine the generated audio file and video file into one AV file (or container file) and include them in the container 590 . The format of the container file stored in the container 590 may be, for example, MP4. However, it is not limited to this format and various formats such as AVI, MP4, MKV, MOV, and ASF.

Referring to FIG. 6 , the camera module 501 includes a first camera 611 (eg, the front camera 405 of FIG. 4A ) and a second camera 612 (eg, the rear camera 412 of FIG. 4B ). The recording module 503 includes a first recorder 621 designated to record video taken by the first camera 611 and a second recorder 622 designated to record video taken by the second camera 612. ) The first recorder 621 and the second recorder 622 generate an AV file including an audio timestamp and a video timestamp in response to a recording command and store the created AV file in a container 590. For example, the processor 599 may process the original image received from the first camera 611 into a first preview image and display it on the display 577. The processor 599 may process the original image received from the first camera 611 into a first preview image and display it on the display 577. The original image received from 612 may be processed into a second preview image and displayed on the display 577. The processor 599 divides the screen of the display 577 to obtain a first preview image and a second preview image. (screen division method) Alternatively, the processor 599 can display one of the first preview image and the second preview image within the other image (picture in picture (PIP) method). While the preview image and the second preview image are displayed, a user's recording command (eg, a touch input to a record button displayed on the touch-sensitive display 577) may be transmitted to the recorders 621 and 622.

According to an embodiment, the first recorder 621 generates video frames (#0, #1, #2, ...) 631 by encoding the video signal in units of frames received from the first camera 611. can do. The first recorder 621 encodes the audio signal in units of frames received from the audio module 502 after the video signal reception from the first camera 611 starts, thereby recording the audio frames #0, #1, #2, …) 632 can be created. The first camera 611 may check the time point t0 from the time information providing module 504 and generate a first video timestamp 633 indicating the checked time point. When the first video frame #0 is generated, the first recorder 621 receives the point of view t0 from the first camera 611 or checks it from the time information providing module 504 and identifies the first video frame #0. Timestamp 633 can be generated. The audio module 502 may check the time point t1 from the time information providing module 504 and generate a first audio timestamp 634 indicating the checked time point. When the first audio frame #0 is generated, the first recorder 621 receives the time point t1 from the audio module 502 or checks it from the time information providing module 504, and the first audio time indicating the checked time point. A stamp 634 may be created. The first recorder 621 may generate and store the first AV file 630 including the data 631 to 634 in the container 590 . The second recorder 622 may generate video frames (#0, #1, #2, ...) 641 by encoding the video signal received from the second camera 612 in units of frames. The second recorder 622 encodes the frame-by-frame audio signal received from the audio module 502 after receiving the video signal from the second camera 612 so that the audio frames #0, #1, #2, …) 642 can be created. The second camera 612 may check the time point t2 from the time information providing module 504 and generate a second video timestamp 643 indicating the checked time point. When the first video frame #0 is generated, the second recorder 622 receives the viewpoint t2 from the second camera 612 or checks it from the time information providing module 504 and confirms the second video frame #0. Timestamp 643 can be generated. The audio module 502 may check the time point t3 from the time information providing module 504 and generate a second audio timestamp 644 indicating the checked time point. When the first audio frame #0 is generated, the second recorder 622 receives the time point t3 from the audio module 502 or checks it from the time information providing module 504, and the second audio time indicating the checked time point. A stamp 644 may be created. The second recorder 622 may generate and store the second AV file 640 including the data 641 to 644 in the container 590 .

The time information providing module 504 converts a numerical value representing time elapsed from a reference time (eg, the time at which a recording command is issued) to a clock in a computing system (eg, the processor 599) of the electronic device 500. It can occur according to the timing of generating the signal. The recorders of the recording module 503 may set the number received from the time information providing module 504 while generating the first frame as a timestamp representing the time when the first frame was created.

As shown in FIG. 6, the AV file (or container file) included in the container 590 includes a header area (or header field), a data area (or data field), and a reserved It can be classified into areas (or preliminary fields). An audio frame and a video frame generated by the recorder may be included in the data area. An audio/video timestamp generated by the recorder may be included in the reserve area. The header area may include meta data related to frames in the data area. For example, the audio meta data may include codec information about a codec used to encrypt and/or decode an audio signal and format information related to a format constituting the audio signal. For example, the audio format information may include bit rate, sampling rate, bit depth, or number of channels. The bit rate is defined as the number of bits processed per unit time (eg, 1 second), and the unit may be bps (bits per second). It can be understood that the higher the bit rate, the higher the quality (eg, fidelity) of the corresponding file. The bit depth corresponds to the resolution of an image, and may be an index indicating how detailed the amplitude of an audio signal can be expressed, and its unit may be bits. As the bit depth increases, the corresponding audio file can be understood with higher resolution. The number of channels may be, for example, 1 (mono), 2 (stereo), 5.1, or 7.1. The sampling rate can be defined as the number of samples (which can be expressed as frames in this document) extracted per unit time (eg, 1 second) from the original sound (eg, an analog audio signal output from a microphone), and The unit may be hertz (Hz). For example, if the sampling rate (which may be expressed as a frame rate in this document) is 48 kHz, there may be 48000 audio frames per second in the audio file. Video meta data may include bit rate and frame rate. The bit rate of video is as defined above, and the frame rate may indicate the number of frames displayed on the screen per unit time (eg, 1 second). For example, when the frame rate is 60 Hz, 60 video frames per second may exist in the corresponding video file.

6 shows two cameras and two recorders, but more cameras and recorders may exist. For example, the electronic device 500 may further include a third camera and a third recorder designated to record video captured by the third camera. In response to the recording command, the third recorder generates an AV file including an audio timestamp and a video timestamp and stores it in the container 590, in the same way as the first recorder 621 and the second recorder 622 do. can be saved

Although not shown, an AV file may include information indicating association with other AV files. For example, the processor 599 may provide association information indicating that the relationship between the first AV file 630 and the second AV file 640 is generated as a recording result for the same recording command (eg, the other party's file name, recording date and time). , a recording mode (eg, director's view)) may be included in each AV file (eg, a header area or a spare area) and stored in the container 590 .

Original AV files (e.g., the first AV file 630 and the second AV file 640) generated as a result of recording for the same recording command may be composed into one AV file synchronized with AV. .

Referring to FIG. 7 , the editing module 505 may select AV files to be synthesized from among AV files included in a container 590 . For example, the editing module 505 may determine the first AV file 630 and the second AV file 640 received from the recorders 621 and 622 through the container 590 as files to be synthesized. As another example, the editing module 505 records the first AV file 630 and the second AV file 640 for the same recording command based on the association information recorded in each AV file included in the container 590. It is recognized that they are generated as a result, and accordingly, the first AV file 630 and the second AV file 640 may be determined as files to be synthesized. The non-synchronization section determination module 506 checks audio/video timestamps in the synthesis target files, and based on the check result, the audio expected to cause AV sync mismatch among the audio/video frames of each synthesis target file. /Can determine video frames (non-synchronized audio/video frames). The editing module 505 uses the remaining frames except for the unsynchronized frames to create video frames (#0, #1, #2, ...) 751 and audio frames (#0, #1, #2, ...). ) 752 may be created and included in the container 590 .

The unsynchronized section determination module 506 checks the video timestamps in the synthesized files, and selects the video timestamp having a relatively large value (eg, generated last in time) as a filter for filtering out the unsynchronized video frames. 1 can be determined as a reference point. As illustrated in FIG. 8 , a second video timestamp t2 having a relatively greater value among the first video timestamp t0 and the second video timestamp t2 may be determined as the first reference point in time. The unsynchronized section determination module 506 checks the audio timestamps in the synthesis target files, and selects the audio timestamp having a relatively large value (eg, generated last in time) as a factor for filtering out the unsynchronized audio frames. 2 can be determined as the reference point. As illustrated in FIG. 8 , among the first audio timestamp t1 and the second audio timestamp t3, the second audio timestamp t3 having a relatively large value may be determined as the second reference time point. Alternatively, the unsynchronized section determination module 506 checks the video timestamp and the audio timestamp in the synthesis target files, and selects a timestamp having a relatively large value (eg, generated last in time) for the unsynchronized video. It may be determined as a reference point in time for filtering out frames and unsynchronized audio frames. As illustrated in FIG. 8 , among the first video timestamp t0, the second video timestamp t1, the first audio timestamp t2, and the second audio timestamp t3, the second audio timestamp ( t3) may be determined as the third reference point in time.

The unsynchronized section determination module 506 may classify video frames generated prior to the first reference point of view or the third reference point of time in the synthesis target files as an asynchronous video frame section to be excluded from video composition. The unsynchronized section determination module 506 may identify video frames generated prior to the first reference point of view or the third reference point of view, based on the frame rate of the corresponding video file. For example, the asynchronous section determination module 506 calculates the time during which one video frame is reproduced as an image through a display (hereinafter referred to as video frame time) based on the frame rate, and from the first (#0) video frame The video frame time can be accumulated up to the nth (#n) video frame. When the accumulated time exceeds the difference value (t2 ? t0) or (t3 ? t0), the first video frame to the n-1 th video frame may be classified as asynchronous video frames. As illustrated in FIG. 8 , the non-synchronized section determination module 506 generates a video frame 631 of the first AV file 630 earlier than the second video timestamp t2 determined as the first reference point in time. The video frames 810 may be classified as unsynchronized video frame intervals. Alternatively, the unsynchronized section determining module 506 determines the video frames 811 generated earlier than the second audio timestamp t3 determined as the third reference point among the video frames 631 of the first AV file 630. may be classified as an asynchronous video frame interval. Also, the non-synchronized section determination module 506 selects video frames 812 generated earlier than the second audio timestamp t3 determined as the third reference point among the video frames 641 of the second AV file 640. It can also be classified as an asynchronous video frame section.

The unsynchronized section determination module 506 may classify an audio frame generated earlier than the second reference point in time (or the third reference point in time) in synthesis target files as an asynchronous audio frame section to be excluded from audio synthesis. The unsynchronized section determination module 506 may identify audio frames generated prior to the second reference point in time based on the sampling rate of the corresponding audio file. For example, the asynchronous section determination module 506 calculates the time during which one audio frame is reproduced as sound through a speaker (hereinafter referred to as audio frame time) based on the sampling rate, and from the first (#0) audio frame Audio frame times may be accumulated up to the mth (#m) audio frame. When the accumulated time exceeds the difference value (t3 − t1), the first audio frame to the m−1 th audio frame may be classified as asynchronous audio frames. As illustrated in FIG. 8 , the non-synchronized section determining module 506 generates an audio frame 632 of the first AV file 630 earlier than the second audio timestamp t3 determined as the second reference point in time. Audio frames may be classified into unsynchronized audio frame intervals 820 . The expression 'interval' exemplified above may be expressed as a period or time. For example, it may be expressed as an asynchronous period instead of an unsynchronized period.

According to one embodiment, the editing module 505 reproduces the video frames in the remaining frame intervals except for the asynchronous video frame interval 810 among the video frames 631 using a codec used for video compression. 1 can be decompressed into a video signal. The editing module 505 may decompress the video frames 641 of the second AV file 640 into a reproducible second video signal using a codec used for video compression. The editing module 505 may obtain one third video signal by combining the first video signal and the second video signal frame by frame. The editing module 505 may generate video frames (#0, #1, #2, ...) 751 by encoding the third video signal frame by frame using a codec used for video decompression. The editing module 505 decompresses the audio frames in the frame section other than the unsynchronized audio frame section 820 from among the audio frames 632 into a first audio signal that can be reproduced using a codec used for audio compression. can do. The editing module 505 may decompress the audio frames 642 of the second AV file 640 into a reproducible second audio signal using a codec used for audio compression or another type of codec. The editing module 505 may obtain one third audio signal by synthesizing the first audio signal and the second audio signal frame by frame. The editing module 505 converts the audio frames (#0, #1, #2, ...) 752 by encoding the third audio signal frame by frame using the codec used for audio decompression or another type of codec. can create The editing module 505 may generate a first composite AV file 750 including video frames 751 and audio frames 752 and store it in a container 590 . Additionally, the editing module 505 generates information representing the time when the first original frames, which are the sources of the first synthesized frame, in the first synthesized AV file 750 are generated, for example, the second video timestamp 643 and the second audio timestamp. 644 may be further included in the spare area of the composite AV file 750 and stored in the container 590. Although not shown, the editing module 505 converts, for example, the file names of the AV files 630 and 640 into the first composite AV file 750 as information representing an original file that is the source of the first composite AV file 750 . It can be stored in the container 590 by further including it in the spare area or header area of the .

According to an embodiment, the editing module 505 may generate a second composite AV file 760 that further includes an asynchronous frame section and maintains synchronization of audio and video. For example, the editing module 505 may decompress the asynchronous video frame section 810 into a reproducible fourth video signal using a codec used for video compression. The editing module 505 generates a fifth video signal by concatenating the fourth video signal and the third video signal in chronological order, and frames the fifth video signal using a codec used in video decompression or another type of codec. The video frames 761 may be generated by encoding in units. The editing module 505 may decompress the unsynchronized audio frame section 820 into a reproducible fourth audio signal using a codec used for audio compression. The editing module 505 generates a fifth audio signal by concatenating the fourth audio signal and the third audio signal in chronological order, encodes the fifth audio signal in frame units using a codec used for audio decompression, and then encodes the audio signal. Frames 762 may be created. The editing module 505 may generate and store the second composite AV file 7650 including the video frames 761 and the audio frames 762 in the container 590 .

The playback module 507 may play an AV file included (or stored) in the container 590 and output it through an output interface. For example, the playback module 507 may decompress the video frames 751 in the first composite AV file 750 into a playable third video signal using a codec used for video compression. The playback module 507 may decompress the audio frames 752 in the first composite AV file 750 into a playable third audio signal using a codec used for audio compression. The playback module 507 may output a third video signal to the display 577 and output a third audio signal to a speaker through the audio module 502 . Accordingly, the third video signal and the third audio signal may be reproduced as images and sounds through the display and the speaker, respectively. As another example, the playback module 507 may decompress the video frames 761 of the second composite AV file 760 into a playable fifth video signal using a codec used for video compression. The reproducing module 507 may decompress the audio frames 762 of the second composite AV file 760 into a reproducible fifth audio signal using a codec used for audio compression. The playback module 507 may output a fifth video signal to the display 577 and output a fifth audio signal to a speaker through the audio module 502 . Accordingly, the fifth video signal and the fifth audio signal may be reproduced as images and sounds through the display and the speaker, respectively.

9 is a flowchart illustrating operations for determining an unsynchronized frame period, according to an exemplary embodiment. The operations of FIG. 9 may be implemented by the processor 599 using the asynchronous section determination module 506 of FIG. 5 or by performing the same function as that configured in the asynchronous section setting module 506 .

In operation 910, the processor 599 checks video timestamps in the video files to be synthesized, and a video timestamp with a relatively large value (eg, generated last in time) and a video timestamp with a relatively small value (eg, generated late in time). For example, a video difference value (or a first difference value) between video timestamps (generated first temporally) may be calculated. The processor 599 checks audio timestamps in the audio files to be synthesized, and determines audio timestamps with a relatively large value (eg, generated last in time) and audio timestamps with a relatively small value (eg, generated last in time). An audio difference value (or a second difference value) between audio timestamps (generated first in terms of time) may be calculated. Alternatively, the processor 599 may calculate a video difference value (or a first difference value) representing a difference between a timestamp having a relatively large value and a video timestamp having a relatively small value among video timestamps and audio timestamps. there is. The processor 599 may calculate an audio difference value (or a second difference value) representing a difference between a timestamp having a relatively large value and an audio timestamp having a relatively small value among video timestamps and audio timestamps. .

In operation 920, the processor 599 may select one of the video files to be synthesized and accumulate video frame times from the first video frame to the n-th video frame in the selected video file. The processor 599 may select one of the audio files to be synthesized and accumulate the audio frame time from the first audio frame to the m-th audio frame in the selected audio file.

In operation 930, the processor 599 may classify the first video frame to the n−1 th video frame as an asynchronous video frame period in the selected video file, based on the video frame accumulation time exceeding the video difference value. The processor 599 may classify the first audio frame to the m−1 th audio frame as an asynchronous audio frame section in the selected audio file, based on the fact that the audio frame accumulation time exceeds the audio difference value.

Operations 920 and 930 may be equally applied to the remaining files to be synthesized.

10 is a flowchart illustrating operations for generating an AV sync-matched composite AV file, according to an embodiment. The operations of FIG. 10 may be implemented by the processor 599 using the modules 503, 504, 505, and 506 of FIG. 5 or by performing the same functions as those configured in the modules 503, 504, 505, and 506. can

In operation 1010, the processor 599 generates a first video file by encoding the video signal received from the first camera of the electronic device 500 frame by frame, and converts the audio signal received from the microphone of the electronic device 500 into frames. A first audio file may be generated by encoding in units, and the first video file and the first audio file may be combined into a first AV file and stored in the memory 588 . While generating the first video file and the first audio file, the processor 599 encodes the video signal received from the second camera of the electronic device 500 frame by frame to generate a second video file, and the electronic device ( A second audio file may be generated by encoding the audio signal received from the microphone of 500 in units of frames, and the second video file and the second audio file may be combined into a second AV file and stored in the memory 588 . When three or more cameras are provided in the electronic device 500, the processor 599 multitasking (eg, the processor 599 simultaneously executes recorders assigned to the cameras respectively) to set three or more cameras. AV files can be created at the same time in the same manner as above.

In operation 1020, the processor 599 generates a first video timestamp indicating when the first frame was created in the first video file of the first AV file and a first audio timestamp indicating when the first frame was created in the first audio file. may be stored in the memory 588. The processor 599 stores a second video timestamp indicating when the first frame was generated in the second video file of the second AV file and a second audio timestamp indicating when the first frame was generated in the second audio file in memory ( 588) can be stored. When three or more AV files are generated at the same time, three or more video/audio timestamps may be stored in the memory 588 in the same manner as described above.

In operation 1030, the processor 599 selects a first timestamp having a relatively large value (eg, indicating a relatively slow time point) among video timestamps included in AV files stored in the memory 588 at the same time. An unsynchronized video frame period (eg, the frame period 810 of FIG. 8) may be determined based on. The processor 599 performs an asynchronous audio frame interval (eg, diagram) based on a second timestamp having a relatively large value (eg, indicating a relatively slow time point) among audio timestamps included in AV files. A frame period 820 of 8) may be determined. For example, the processor 599 may set the largest value among video timestamps as a reference view, and determine video frames generated prior to the reference view as frames belonging to an asynchronous video frame period. The processor 599 may set the largest value among the audio timestamps as a reference time, and determine audio frames generated prior to the reference time as frames belonging to the asynchronous audio frame period. In another embodiment, the processor 599 determines the asynchronous video frame period by using the timestamp having the largest value among video timestamps and audio timestamps included in AV files stored at the same time in the memory 588. may be selected as the first timestamp used for Also, the processor 599 may select a timestamp having the largest value among the video timestamps and the audio timestamps as the second timestamp used to determine the unsynchronized audio frame period.

In operation 1040, the processor 599 may generate a plurality of video signals by decoding video frames that do not belong to an asynchronous video frame period for each AV file. The processor 599 may generate a plurality of audio signals by decoding audio frames that do not belong to an asynchronous audio frame period for each AV file.

In operation 1050, the processor 599 synthesizes a plurality of video signals into one video signal and encodes them in units of frames to generate a single composite video file, and synthesizes a plurality of audio signals into one audio signal and encodes them in units of frames. By doing so, a single synthesized audio file can be created. The processor 599 may combine the composite video file and the composite audio file into a single composite AV file and store it in the memory 588 .

In operation 1060, the processor 599 may reproduce and output the composite AV file through a display and a speaker. Additionally, the processor 599 may play the audio/video frames in the asynchronous frame period first and then play the composite AV file. Referring to FIG. 8 , the processor 599 may reproduce the audio/video frames within the asynchronous frame intervals 810 and 820 and then reproduce the first composite AV file 750 .

In various embodiments, an electronic device (eg, the electronic device 500 of FIG. 5 ) includes cameras; mike; a processor operatively connected to the cameras and the microphone; and a memory operatively coupled to the processor. When the memory (eg, memory 588 of FIG. 5 ) is executed, the processor (eg, processor 599 of FIG. 5 ): encodes a video signal received from a first one of the cameras in units of frames. To generate a first video file, to generate a first audio file by encoding the audio signal received from the microphone frame by frame, to combine the first video file and the first audio file into a first AV file, storing in a memory, and generating a second video file by encoding a video signal received from a second camera of the cameras frame by frame at the same time period during which the first video file and the first audio file are generated; A second audio file is generated by encoding the audio signal received from the microphone frame by frame, the second video file and the second audio file are combined into a second AV file, stored in the memory, and the first video file A first video timestamp indicating when the first frame was created in , a first audio timestamp indicating when the first frame was created in the first audio file, and a second video timestamp indicating when the first frame was created in the second video file. 2 A video timestamp and a second audio timestamp indicating a time when a first frame was generated in the second audio file are stored in the memory, and among video timestamps included in AV files stored in the memory at the same time period, or determining an unsynchronized video frame period based on a first timestamp indicating a relatively slow time point among video timestamps and audio timestamps included in the AV files, and determining an asynchronous video frame period, and Determines an unsynchronized audio frame period based on a second timestamp representing a relatively slow time point among video timestamps and audio timestamps included in AV files or among video timestamps and audio timestamps included in AV files, and determines an unsynchronized video frame interval for each AV file. A plurality of video signals are generated by decoding video frames not belonging to a section, the plurality of video signals are synthesized into one video signal, and a composite video file is generated by encoding the synthesized video signal, and the AV file A plurality of audio signals are generated by decoding audio frames that do not belong to the unsynchronized audio frame interval for each segment, the plurality of audio signals are synthesized into one audio signal, and the synthesized audio signal is encoded to generate a synthesized audio file. and storing instructions for generating and combining the composite video file and the composite audio file into a composite AV file and storing the composite AV file in the memory.

The instructions include the processor determining the first timestamp as a video reference time, determining a video frame generated before the video reference time as a video frame within the asynchronous video frame period, and determining the second timestamp as an audio reference time. An audio frame generated prior to the audio reference point in time determined as the reference point in time may be determined as an audio frame within the asynchronous audio frame period.

In the instructions, when the processor identifies an audio frame generated prior to the audio reference viewpoint and a video frame generated prior to the video reference viewpoint, one video frame from the first video frame to the n-th video frame is displayed through the display. Accumulate video frame times reproduced as images, and if the accumulated video frame times exceed the difference between the largest value and the smallest value among the video timestamps, from the first video frame to the n-1th video frame. determined as the asynchronous video frame period, and accumulating audio frame times at which one audio frame is reproduced as sound through a speaker from the first audio frame to the n-th audio frame, and the accumulated audio frame times correspond to the audio timestamps When the difference value between the largest value and the smallest value is exceeded, the asynchronous audio frame section may be determined from the first audio frame to the n−1 th audio frame.

The instructions may cause the processor to reproduce the synthesized AV file and output images and sounds through a display and a speaker, and reproduce the asynchronous video frame section and the asynchronous audio frame section before the synthesized AV file. there is.

The first camera may be a camera disposed on a front side of the electronic device, and the second camera may be a camera disposed on a rear side of the electronic device.

The instructions cause the processor to store the first audio timestamp and the first video timestamp in a reserved area of the first AV file, and to store the second audio timestamp and the second video timestamp in the second AV file. It can be stored in the reserved area of the AV file.

The memory may include a container including an AV file, and the container may be an MP4 file.

The instructions may cause the processor to generate a video file by compressing a video signal using a video codec associated with the MP4 container, and to generate an audio file by compressing an audio signal using an audio codec associated with the MP4 container. there is.

The instructions may cause the processor to generate an audio timestamp and a video timestamp based on a system clock generated by the processor.

A plurality of recorders configured to correspond to the cameras one-to-one and record video captured by the corresponding cameras are stored in the memory, and the instructions are configured so that the processor responds to a recording command received from an input device of the electronic device, By simultaneously executing the recorders, a plurality of AV files can be created at the same time.

In various embodiments, a method of operating an electronic device (eg, the electronic device 500 of FIG. 5 ) encodes a video signal received from a first camera among cameras included in the electronic device in units of frames to generate a first video signal. A file is generated, an audio signal received from the microphone is encoded frame by frame to generate a first audio file, the first video file and the first audio file are combined into a first AV file, and the memory of the electronic device is generated. operation of storing in (eg, operation 1010 of FIG. 10); A second video file is generated by encoding a video signal received from a second camera among the cameras frame by frame during the same time period in which the first video file and the first audio file are generated, and the audio received from the microphone is generated. generating a second audio file by encoding a signal frame by frame, combining the second video file and the second audio file into a second AV file, and storing the second audio file in the memory (eg, operation 1010 of FIG. 10 ); A first video timestamp indicating when the first frame was created in the first video file, a first audio timestamp indicating when the first frame was created in the first audio file, and the first frame in the second video file storing in the memory a second video timestamp indicating when a first frame was created and a second audio timestamp indicating when a first frame was created in the second audio file (eg, operation 1020 of FIG. 5 ); Based on a first timestamp indicating a relatively slow time point among video timestamps included in AV files stored in the memory at the same time, or among video timestamps and audio timestamps included in the AV files, the ratio Determines a synchronization video frame interval, based on a second timestamp indicating a relatively slow time point among audio timestamps included in the AV files or among video timestamps and audio timestamps included in the AV files to determine an unsynchronized audio frame section (eg, operation 1030 of FIG. 5); For each of the AV files, a plurality of video signals are generated by decoding video frames that do not belong to an asynchronous video frame period, the plurality of video signals are synthesized into one video signal, and the video signal synthesized into one is synthesized by encoding. creating a video file (eg, operations 1040 and 1050 of FIG. 5 ); For each of the AV files, a plurality of audio signals are generated by decoding audio frames that do not belong to the asynchronous audio frame period, the plurality of audio signals are synthesized into one audio signal, and the synthesized audio signal is synthesized by encoding. creating an audio file (eg, operations 1040 and 1050 of FIG. 5 ); and combining the composite video file and the composite audio file into a composite AV file and storing the composite AV file in the memory (eg, operation 1050 of FIG. 5 ).

The determining of the unsynchronized video frame period and the unsynchronized audio frame period may include determining the first timestamp as a video reference time and using a video frame generated before the video reference time as a video within the asynchronous video frame period. and determining the second timestamp as an audio reference time point and determining an audio frame generated prior to the audio reference time point as an audio frame within the asynchronous audio frame section.

The determining of the unsynchronized video frame interval may include accumulating video frame times at which one video frame is reproduced as an image through a display from a first video frame to an n-th video frame, and the accumulated video frame time corresponds to the video frame time. and determining the asynchronous video frame interval from the first video frame to the n−1 th video frame when the difference between the largest value and the smallest value among the timestamps is exceeded. The determining of the unsynchronized audio frame section may include accumulating audio frame times in which one audio frame is reproduced as sound through a speaker from a first audio frame to an n-th audio frame, and the accumulated audio frame time is the audio frame time. and determining the asynchronous audio frame period from the first audio frame to the n−1 th audio frame when the difference between the largest value and the smallest value among the timestamps is exceeded.

The method reproduces the synthesized AV file and outputs images and sounds through a display and a speaker, and reproduces the asynchronous video frame section and the asynchronous audio frame section before the synthesized AV file (eg, FIG. 5 ). Operation 1060 of) may be further included.

The storing of the first AV file and the second AV file may include: storing the first audio timestamp and the first video timestamp in a spare area of the first AV file; and storing the stamp and the timestamp of the second video in a reserved area of the second AV file.

The embodiments of the present invention disclosed in the present specification and drawings are only presented as specific examples to easily explain the technical content according to the embodiments of the present invention and help understanding of the embodiments of the present invention, and limit the scope of the embodiments of the present invention. It's not what I want to do. Therefore, the scope of various embodiments of the present invention should be construed as including all changes or modified forms derived based on the technical spirit of various embodiments of the present invention in addition to the embodiments disclosed herein are included in the scope of various embodiments of the present invention. .

Claims (15)

In electronic devices,
cameras;
mike;
a processor operatively coupled to the cameras and the microphone; and
a memory operatively coupled to the processor;
The memory, when executed, causes the processor to:
A first video file is generated by encoding a video signal received from a first camera of the cameras in units of frames, and a first audio file is generated by encoding an audio signal received from the microphone in units of frames. combining a video file and the first audio file into a first AV file and storing the first AV file in the memory;
A second video file is generated by encoding a video signal received from a second camera among the cameras frame by frame during the same time period in which the first video file and the first audio file are generated, and the audio received from the microphone is generated. generating a second audio file by encoding a signal frame by frame, combining the second video file and the second audio file into a second AV file, and storing the second audio file in the memory;
A first video timestamp indicating when the first frame was created in the first video file, a first audio timestamp indicating when the first frame was created in the first audio file, and the first frame in the second video file storing in the memory a second video timestamp indicating a time when a first frame was created and a second audio timestamp indicating a time when a first frame was generated in the second audio file;
Based on a first timestamp indicating a relatively slow time point among video timestamps included in AV files stored in the memory at the same time, or among video timestamps and audio timestamps included in the AV files, the ratio determining a synchronization video frame interval;
Determining an unsynchronized audio frame interval based on a second timestamp indicating a relatively slow time point among audio timestamps included in the AV files or among video timestamps and audio timestamps included in the AV files; ,
For each of the AV files, a plurality of video signals are generated by decoding video frames that do not belong to an asynchronous video frame period, the plurality of video signals are synthesized into one video signal, and the video signal synthesized into one is synthesized by encoding. create a video file;
For each of the AV files, a plurality of audio signals are generated by decoding audio frames that do not belong to the asynchronous audio frame period, the plurality of audio signals are synthesized into one audio signal, and the synthesized audio signal is synthesized by encoding. create an audio file;
An electronic device storing instructions for combining the composite video file and the composite audio file into a composite AV file and storing the composite AV file in the memory. The method of claim 1, wherein the instructions are the processor,
determining the first timestamp as a video reference viewpoint and determining a video frame generated before the video reference viewpoint as a video frame within the asynchronous video frame interval;
The electronic device determines the second timestamp as an audio reference time and determines an audio frame generated before the audio reference time as an audio frame within the asynchronous audio frame section. 3. The method of claim 2, wherein the instructions are configured by the processor to identify an audio frame generated prior to the audio reference viewpoint and a video frame generated prior to the video reference viewpoint,
From the first video frame to the n-th video frame, video frame times during which one video frame is reproduced as an image through a display are accumulated, and the accumulated video frame time is the difference between the largest value and the smallest value among the video timestamps. If it exceeds the value, determining the asynchronous video frame interval from the first video frame to the n-1 th video frame;
One audio frame from the first audio frame to the n-th audio frame accumulates audio frame times reproduced as sound through a speaker, and the accumulated audio frame time is the difference between the largest value and the smallest value among the audio timestamps. If the value exceeds the value, the electronic device determines the asynchronous audio frame period from the first audio frame to the n-1 th audio frame. The method of claim 1, wherein the instructions are the processor,
An electronic device that reproduces the synthesized AV file and outputs images and sounds through a display and a speaker, wherein the asynchronous video frame section and the asynchronous audio frame section are played before the synthesized AV file. According to claim 1,
The first camera is a camera disposed on the front of the electronic device,
The second camera is a camera disposed on a rear surface of the electronic device. The method of claim 1, wherein the instructions are the processor,
store the first audio timestamp and the first video timestamp in a reserved area of the first AV file;
and storing the second audio timestamp and the second video timestamp in a reserved area of the second AV file. The method of claim 1, wherein the memory comprises a container including an AV file,
An electronic device in which the container is MP4. The method of claim 7, wherein the instructions are the processor,
generating a video file by compressing a video signal using a video codec associated with the MP4 container;
An electronic device for generating an audio file by compressing an audio signal using an audio codec associated with the MP4 container. The method of claim 1, wherein the instructions are the processor,
An electronic device that generates an audio timestamp and a video timestamp based on a system clock generated by the processor. The method of claim 1 , wherein a plurality of recorders configured to correspond to the cameras one-to-one and record video captured by the corresponding cameras are stored in the memory, wherein the instructions are configured to:
An electronic device that generates a plurality of AV files at the same time by simultaneously executing the recorders in response to a recording command received from an input device of the electronic device. A method for operating an electronic device,
A first video file is generated by encoding a video signal received from a first camera among cameras included in the electronic device in units of frames, and a first audio file is generated by encoding an audio signal received from the microphone in units of frames. combining the first video file and the first audio file into a first AV file and storing the combined first AV file in a memory of the electronic device;
A second video file is generated by encoding a video signal received from a second camera among the cameras frame by frame during the same time period in which the first video file and the first audio file are generated, and the audio received from the microphone is generated. generating a second audio file by encoding a signal frame by frame, combining the second video file and the second audio file into a second AV file, and storing the second audio file in the memory;
A first video timestamp indicating when the first frame was created in the first video file, a first audio timestamp indicating when the first frame was created in the first audio file, and the first frame in the second video file storing, in the memory, a second video timestamp indicating when a first frame was created and a second audio timestamp indicating when a first frame was generated in the second audio file;
Based on a first timestamp indicating a relatively slow time point among video timestamps included in AV files stored in the memory at the same time, or among video timestamps and audio timestamps included in the AV files, the ratio Determines a synchronization video frame interval, based on a second timestamp indicating a relatively slow time point among audio timestamps included in the AV files or among video timestamps and audio timestamps included in the AV files determining an unsynchronized audio frame section by performing the step;
For each of the AV files, a plurality of video signals are generated by decoding video frames that do not belong to an asynchronous video frame period, the plurality of video signals are synthesized into one video signal, and the video signal synthesized into one is synthesized by encoding. creating a video file;
For each of the AV files, a plurality of audio signals are generated by decoding audio frames that do not belong to the asynchronous audio frame period, the plurality of audio signals are synthesized into one audio signal, and the synthesized audio signal is synthesized by encoding. creating an audio file; and
and combining the composite video file and the composite audio file into a composite AV file and storing the composite AV file in the memory. The method of claim 11, wherein determining the asynchronous video frame period and the asynchronous audio frame period comprises:
determining the first timestamp as a video reference viewpoint and determining a video frame generated before the video reference viewpoint as a video frame within the asynchronous video frame period;
and determining the second timestamp as an audio reference time point and determining an audio frame generated prior to the audio reference time point as an audio frame within the asynchronous audio frame period. According to claim 12,
The operation of determining the asynchronous video frame period
accumulating video frame times during which one video frame is reproduced as an image through a display from the first video frame to the nth video frame;
determining the asynchronous video frame period from the first video frame to the n-1 th video frame when the accumulated video frame time exceeds the difference between the largest value and the smallest value among the video timestamps; include,
The operation of determining the unsynchronized audio frame period
accumulating audio frame times during which one audio frame is reproduced as sound through a speaker from the first audio frame to the nth audio frame;
determining the asynchronous audio frame period from the first audio frame to the n−1 th audio frame when the accumulated audio frame time exceeds a difference between the largest value and the smallest value among the audio timestamps; How to include. According to claim 11,
The method of claim 1 , further comprising reproducing the synthesized AV file and outputting images and sounds through a display and a speaker, and reproducing the asynchronous video frame section and the asynchronous audio frame section prior to the synthesized AV file. 12. The method of claim 11, wherein the storing of the first AV file and the second AV file comprises:
storing the first audio timestamp and the first video timestamp in a reserved area of the first AV file;
and storing the second audio timestamp and the second video timestamp in a reserved area of the second AV file.

Images