CN102487439B - Audio and video acquisition and play processing method with whole embedding of memory - Google Patents

Audio and video acquisition and play processing method with whole embedding of memory Download PDF

Info

Publication number
CN102487439B
CN102487439B CN201010568484.1A CN201010568484A CN102487439B CN 102487439 B CN102487439 B CN 102487439B CN 201010568484 A CN201010568484 A CN 201010568484A CN 102487439 B CN102487439 B CN 102487439B
Authority
CN
China
Prior art keywords
video
data
memory
internal memory
audio
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201010568484.1A
Other languages
Chinese (zh)
Other versions
CN102487439A (en
Inventor
陈智德
李钊辉
贾权
黄宇浩
杜林峰
李小明
胡胜发
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Ankai Microelectronics Co.,Ltd.
Original Assignee
Anyka Guangzhou Microelectronics Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Anyka Guangzhou Microelectronics Technology Co Ltd filed Critical Anyka Guangzhou Microelectronics Technology Co Ltd
Priority to CN201010568484.1A priority Critical patent/CN102487439B/en
Publication of CN102487439A publication Critical patent/CN102487439A/en
Application granted granted Critical
Publication of CN102487439B publication Critical patent/CN102487439B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Abstract

The invention discloses an audio and video acquisition and play processing method with the whole embedding of a memory. An audio and video data acquisition process comprises the following steps: (A) coding audio data and writing the data into an external memory; (B) processing video data by employing a hardware acceleration method. An audio and video playing process comprises the following steps: (I) system software reads a code stream file from the external memory, completes parse work of a file header, and separates an audio stream and a video stream, and audio stream data is subjected to decoding playing by using an CPU; (II) video stream data is processed by employing the hardware acceleration method and is played. According to the method, in a condition that a multimedia application processor does not need to connect a DRAM externally or other external random memory cells, through only depending on an on-chip memory and employing the hardware acceleration method, common functions of real-time audio and video recording and playing and the like are completed, cost is low, and popularization utilization is facilitated.

Description

Audio-video collection and play handling method that a kind of internal memory all embeds
Technical field
The present invention relates to the audio/video processing method in a kind of embedded system, relate in particular to audio-video collection and play handling method that a kind of internal memory all embeds.
Background technology
Along with the continuous growth of the development of multimedia chip technology and the application market of association area, the audio frequency and video disposal ability of multimedia application processor is more and more stronger.The factor such as improve constantly of the demand of real-time, coding/decoding capability, makes audio video processor toward hardware-accelerated future development, and often needs the assistance of external internal memory.Now the audio frequency and video treatment circuit of main flow mainly contains two kinds: a kind of is to complete the functions such as the recording of audio frequency and video, broadcasting by the method for pure software, and external internal memory is for CPU; Be to have assisted the functions such as the recording of audio frequency and video, broadcasting, most data processing to adopt hardware logic to realize by hardware-accelerated method, external internal memory for CPU and hardware-accelerated unit jointly.
First method completes the functions such as the recording of audio frequency and video, broadcasting by the method for pure software, and external internal memory is for CPU.The method of pure software is very painstaking relating to that the data volumes such as Video processing are large, in the processing of complicated operation.The requirement of real-time and the raising of coding/decoding capability (being mainly reflected in improving constantly of video resolution), require the ability of CPU core to improve constantly, dominant frequency also needs continuous lifting, finally causes cost to continue to increase.CPU running has at a high speed determined that its power consumption is very high, has become fatal shortcoming in portable application.
Second method has been assisted the functions such as the recording of audio frequency and video, broadcasting by hardware-accelerated method.Hardware-accelerated unit has alleviated the burden of CPU greatly, therefore becomes the main flow of current high-resolution video processor.But hardware-accelerated unit needs directly access to use a large amount of memory cell in computing, therefore need external memory chip (as DRAM) for it as buffer memory.External memory chip performance is higher, and memory cell is larger, and complete machine cost is just higher.
Summary of the invention
The audio-video collection and the play handling method that the object of the present invention is to provide a kind of internal memory all to embed, can allow multimedia application processor without external DRAM or other random external memory cell in the situation that, only rely on sheet and store, adopt hardware-accelerated method, complete the common functions such as real-time audio frequency and video are recorded, broadcasting, and cost is lower, contribute to promote the use.
Object of the present invention can realize by following technical measures:
The audio-video collection processing method that internal memory all embeds, comprises the following steps:
(A) voice data encoded and write in external memory storage;
(B) adopt hardware-accelerated method to process to video data.
The hardware-accelerated processing procedure of described video data is:
B1), by camera collection to video data be buffered on sheet in internal memory BUF1, encoder reads source data and carries out Video coding from sheet in internal memory BUF1;
B2), completed after a frame coding, CPU, by external memory interface, writes ASCII stream file ASCII in external memory storage.
Described step B1) middle camera collection is with pixel behavior unit, and Video coding is taking macro block as unit, and each macro block comprises 16 pixel columns or 8 pixel columns conventionally.
Described step B1) in sheet on internal memory BUF1 be the ping-pong buffer of the data of 2 macro-block line of buffer memory.
Described step B1) in the gatherer process of encoder and camera adopt flowing water parallel processing mode, detailed process is: when the gatherer process of encoder and camera has all completed after macro-block line data processing separately, exchange ping-pong buffer, continues the processing of next macro-block line mutually.
At described step B2) in Video coding, carry out the process of playing in real time: transmit control module obtains the data of current collection from sheet in internal memory BUF1, and zoom to the size that is applicable to showing and deliver to external display and show.
Described external display or its driving chip have frame buffer.
The audio frequency and video play handling method that the present invention is a kind of and the matching used internal memory of above-mentioned acquisition method all embeds, following steps:
(I), systems soft ware readout code stream file from external memory storage, and complete the parsing work of file header, isolate audio stream and video flowing; The CPU broadcasting of decoding for described audio stream data;
(II), described video stream data adopts hardware-accelerated processing and plays.
In described step (II), video data adopts the detailed process that hardware-accelerated method is processed and play to be: CPU handles after frame head data, the data flow of one frame is left on sheet in internal memory BUF2, decoder reading out data in internal memory BUF2 from sheet, and complete decoding, and by decoded deposit data on sheet in internal memory BUF1; On sheet, internal memory BUF1 receives after the data of a macro-block line, sends into transmit control module and plays.
On sheet in described embedding internal memory, on internal memory BUF1 and sheet, internal memory BUF2 adopts multiplex mode, and described multiplex mode comprises: Video processing pattern and Installed System Memory pattern; Wherein, Video processing pattern is switched to the process of Installed System Memory pattern and is:
A1), mode of operation is switched to Installed System Memory pattern;
B1), be current Memory Allocation address, and be set to available;
C1), CPU normally accesses current memory headroom;
The process that Installed System Memory pattern is switched to Video processing pattern is:
A2), check whether current internal memory is using, if the current internal memory free time directly enters step c2); Otherwise enter step b2);
B2), by the content replication of current internal memory in other available memory space;
C2), mode of operation is switched to the hardware-accelerated tupe of video data.
The inventive method has following beneficial effect compared to existing technology:
1, on the one hand, existingly carry out coding and decoding video with pure software, the slow shortcoming of arithmetic speed makes it be difficult to reach the requirement of real-time; The CPU high power consumption causing that runs up for a long time, makes it be difficult to meet portable needs.By contrast, the inventive method has adopted specialized hardware accelerator module, completes the most of data operation in coding and decoding video, can meet the requirement of real-time in speed; On the other hand, the processor active task of CPU alleviates greatly, only need to complete file organization, and the simple work such as file header parsing has greater advantage in power consumption;
2,, compared with the scheme that participates in Video processing with external internal memory, the inventive method can significantly reduce costs.The scheme of external internal memory conventionally by the data of Video processing taking frame as unit is cached in outside DRAM in, and in the present invention, because the correlation of codec data is little, only need the data of 2 macro-block line of buffer memory, the buffer memory of this order of magnitude can be realized completely in sheet.By contrast, the present invention, without external internal memory, effectively reduces complete machine cost; On the other hand, adopt processor chips of the present invention without encapsulating out a large amount of pins in external Memory linkage, can effectively reduce the packaging cost of chip.Meanwhile, because storage resources access speed on sheet is fast, be with widely, a lot of data times of saving compared with external internal memory, make the average encoding and decoding time increase.
Brief description of the drawings
Fig. 1 is the schematic flow sheet of the audio, video data gatherer process of the inventive method;
Fig. 2 is the data interaction schematic diagram of camera collection and video encoder in the inventive method audio, video data gatherer process;
Fig. 3 is the schematic flow sheet of the playing process of the audio frequency and video of the inventive method;
Fig. 4 is the switching flow figure that embeds internal memory in the inventive method and be converted to from Video processing pattern Installed System Memory pattern;
Fig. 5 is the switching flow figure that embeds internal memory in the inventive method and be converted to from Installed System Memory pattern Video processing pattern.
Embodiment
Fig. 1 to Fig. 5 shows all each several part handling processes of the audio video processor inside of embedding of internal memory of the present invention, and this audio frequency and video handling process comprises the replayed section of audio, video data gatherer process and video, and wherein, audio, video data gatherer process is:
(A) voice data encoded and write in external memory storage;
(B) adopt hardware-accelerated method to process to video data;
The hardware-accelerated processing procedure of video data is wherein:
B1), by camera collection to video data be buffered on sheet in internal memory BUF1, encoder reads source data and carries out Video coding from BUF1; Camera collection is with pixel behavior unit, and Video coding is taking macro block as unit, and each macro block comprises 16 pixel columns or 8 pixel columns conventionally.On sheet, internal memory BUF1 is the ping-pong buffer of the data of 2 macro-block line of buffer memory, realizes in RAM mode on sheet.
The gatherer process of encoder and camera adopts flowing water parallel processing mode, and detailed process is: when the gatherer process of encoder and camera has all completed after macro-block line data processing separately, mutually exchange ping-pong buffer, continue the processing of next macro-block line.In general SD camera application (as 720x480@25fps), the coder processes time is faster than the camera collection time, and this has ensured that image data can not be lost endlessly, and Video coding can meet the requirement of real-time.Data interactive method between camera and encoder is with macro block behavior unit ping-pong buffer, and this is the basic reason of greatly dwindling Video coding spatial cache, is also that Video processing can be accomplished the key point that complete internal memory embeds.At step B1) complete before a frame coding, also insert real-time playing process: transmit control module obtains the data of current collection from BUF1, and zoom to the size that is applicable to showing and deliver to external display and show.External display or its driving chip all have frame buffer function.
At step B1) complete before a frame coding, also insert real-time playing process: transmit control module obtains the data of current collection from BUF1, and zoom to the size that is applicable to showing and deliver to external display and show.External display or its driving chip all have frame buffer function.
Encoder adopts intraframe coding technology, to reduce the dependence to non-current frame data.A kind of typical case is applied as JPEG compression standard (but being not limited only to JPEG compression standard).Completed after coding, code stream is stored in code stream buffer memory BUF2.
B2), completed after a frame coding, CPU, by external memory interface, writes ASCII stream file ASCII in external memory storage, as Nand Flash, SD card etc.;
In addition, the detailed process of the replayed section of matching used audio frequency and video is:
(I), systems soft ware readout code stream file from external memory storage, and complete the parsing work of file header, isolate audio stream and video flowing; The CPU broadcasting of decoding for described audio stream data;
(II), video stream data adopts hardware-accelerated method to process and play.The detailed process of playing is: CPU handles after frame head data, the data flow of a frame is left on sheet in internal memory BUF2, decoder reading out data in internal memory BUF2 from sheet, and complete decoding, and by decoded deposit data on sheet in internal memory BUF1; On sheet, internal memory BUF1 receives after the data of a macro-block line, sends into transmit control module and plays
Wherein, the BUF1 in above-mentioned embedding internal memory and BUF2 adopt multiplex mode, and multiplex mode comprises: Video processing pattern and Installed System Memory pattern.In the time not needing the function of Video processing in application, BUF1 and BUF2 can be multiplexed into Installed System Memory for CPU, make system have more abundant internal memory to process other business.Switching between two kinds of patterns is carried out according to the flow process of Fig. 4 and Fig. 5.Wherein, Video processing pattern is switched to the process of Installed System Memory pattern and is:
A1), mode of operation is switched to Installed System Memory pattern;
B1), be current Memory Allocation address, and be set to available;
C1), CPU normally accesses current memory headroom, realize in RAM mode on sheet equally;
The process that Installed System Memory pattern is switched to Video processing pattern is:
A2), check whether current internal memory is using, if the current internal memory free time directly enters step c2); Otherwise enter step b2);
B2), by the content replication of current internal memory in other available memory space;
C2), mode of operation is switched to the hardware-accelerated processing procedure of video data
The inventive method can be applied to mobile monitor field (as video pen), also can be applied to portable multimedia apparatus (as the PMP with camera).
Taking a PMP with camera as example, if require to support the camera of maximum 300,000 pixels, the video recording frame per second of 640x480@25fps, support the LCD screen of QVGA (320x240) with MPU interface, the audio frequency and video treatment circuit that adopts the internal memory in the present invention all to embed, the index of all parts in circuit is as follows:
Be 40ms/f the average time of a, camera collection;
B, BUF1 need to deposit 2 macro-block line, i.e. the data of 32 pixel columns are calculated according to the form of YUV420, and BUF1 needs on the sheet of 640x32x1.5=30KB size RAM as buffer memory;
C, video encoder need to meet the average acquisition time that the average scramble time is less than camera, adopt JPEG compression standard to meet the demands, and it is greatly about 20ms/f left and right;
D, BUF2 are set to hold by experience the size that exceedes 1 frame coded data and are advisable, and are set as 24KB;
E, playing controller can complete 640x480 and dwindle demand to the resolution of 320x240, and it shows that required time is far smaller than the scramble time of video encoder.
Comprehensively above-mentioned, in this example, adopt the circuit in the present invention, spatial cache that only need to about 54KB left and right, can realize video record and the playing task of 640x480@25fps.This big or small storage resources is to be applicable to realizing with embedding internal memory on sheet completely.Its direct advantage is just to have saved the cost of external internal memory, has reduced the costs such as the encapsulation of chip.
Embodiments of the present invention are not limited to this, and in the present invention, typical case is applied as data that utilizing camera interface collects and sends into and in BUF1, carry out buffer memory.Before entering encoder, also can first carry out some pre-treatment work; as the embedding of convergent-divergent processing, timestamp or other guide; colour switching; window intercepting etc.; but be all to have adopted the data buffer storage processing method of carrying out with macro block behavior unit in the present invention in essence; finally realize the object that internal memory all embeds, within therefore also should being considered as the protection range of right of the present invention.
In some applications, some main modular can be default.Such as some mobile monitor product may not need live preview and Presentation Function, playing controller can be default.But if the hardware handles technology that its main part has adopted the internal memory in the present invention all to embed, within also should being considered as the protection range of right of the present invention.Therefore,, under the above-mentioned basic fundamental thought of the present invention prerequisite, amendment, replacement or the change to other various ways that content of the present invention is made according to the ordinary skill knowledge of this area and customary means, within all dropping on rights protection scope of the present invention.

Claims (1)

1. the audio frequency and video play handling method that internal memory all embeds, is characterized in that comprising:
(I), systems soft ware readout code stream file from external memory storage, and complete the parsing work of file header,
Isolate audio stream and video flowing; The CPU broadcasting of decoding for described audio stream data;
(II), described video stream data adopts hardware-accelerated method to process and play;
In described step (II), video data adopts the detailed process that hardware-accelerated method is processed and play to be: CPU handles after frame head data, the data flow of one frame is left on sheet in internal memory BUF2, decoder reading out data in internal memory BUF2 from sheet, and complete decoding, and by decoded deposit data on sheet in internal memory BUF1; On sheet, internal memory BUF1 receives after the data of a macro-block line, sends into transmit control module and plays;
On described, on internal memory BUF1 and sheet, internal memory BUF2 adopts multiplex mode, and described multiplex mode comprises: Video processing pattern and Installed System Memory pattern; Wherein, Video processing pattern is switched to the process of Installed System Memory pattern and is:
A1), mode of operation is switched to Installed System Memory pattern;
B1), be current Memory Allocation address, and be set to available;
C1), CPU normally accesses current memory headroom;
The process that Installed System Memory pattern is switched to Video processing pattern is:
A2), check whether current internal memory is using, if the current internal memory free time directly enters step c2); Otherwise enter step b2);
B2), by the content replication of current internal memory in other available memory space;
C2), mode of operation is switched to the hardware-accelerated tupe of video data.
CN201010568484.1A 2010-12-01 2010-12-01 Audio and video acquisition and play processing method with whole embedding of memory Active CN102487439B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201010568484.1A CN102487439B (en) 2010-12-01 2010-12-01 Audio and video acquisition and play processing method with whole embedding of memory

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201010568484.1A CN102487439B (en) 2010-12-01 2010-12-01 Audio and video acquisition and play processing method with whole embedding of memory

Publications (2)

Publication Number Publication Date
CN102487439A CN102487439A (en) 2012-06-06
CN102487439B true CN102487439B (en) 2014-12-10

Family

ID=46152889

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201010568484.1A Active CN102487439B (en) 2010-12-01 2010-12-01 Audio and video acquisition and play processing method with whole embedding of memory

Country Status (1)

Country Link
CN (1) CN102487439B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107613302B (en) * 2017-09-13 2020-10-02 珠海格力电器股份有限公司 Decoding method and device, storage medium and processor
CN108174283A (en) * 2017-12-27 2018-06-15 威创集团股份有限公司 A kind of vision signal source generating method and device
CN108282696A (en) * 2018-02-07 2018-07-13 北京易讯理想科技有限公司 A kind of hardware resource distribution method that sequence frame image plays

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5774676A (en) * 1995-10-03 1998-06-30 S3, Incorporated Method and apparatus for decompression of MPEG compressed data in a computer system
CN101009836A (en) * 2007-01-31 2007-08-01 浙江大学 Embedded video playing device based on the dual processor
CN101383954A (en) * 2007-09-06 2009-03-11 北京中电华大电子设计有限责任公司 Implementing method for media processing chip supporting multiple audio and video standard
CN101742218A (en) * 2010-01-19 2010-06-16 南京邮电大学 Implementation method of videophone system among IPTV set top boxes

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5774676A (en) * 1995-10-03 1998-06-30 S3, Incorporated Method and apparatus for decompression of MPEG compressed data in a computer system
CN101009836A (en) * 2007-01-31 2007-08-01 浙江大学 Embedded video playing device based on the dual processor
CN101383954A (en) * 2007-09-06 2009-03-11 北京中电华大电子设计有限责任公司 Implementing method for media processing chip supporting multiple audio and video standard
CN101742218A (en) * 2010-01-19 2010-06-16 南京邮电大学 Implementation method of videophone system among IPTV set top boxes

Also Published As

Publication number Publication date
CN102487439A (en) 2012-06-06

Similar Documents

Publication Publication Date Title
CN105120293B (en) Image collaboration coding/decoding method and device based on CPU and GPU
CN101252694B (en) Address mapping system and frame storage compression of video frequency decoding based on blocks
CN105993177A (en) Image capture accelerator
CN102724492B (en) Method and system for transmitting and playing video images
CN103442204A (en) Network video transmission system and method based on DM365
CN105208342B (en) A kind of two-way video compression storage and network switching transmission circuit
CN102487439B (en) Audio and video acquisition and play processing method with whole embedding of memory
CN106412588A (en) Image frame processing method
CN106412473A (en) Image processing device, movie subsystem and movie processing circuit
Li et al. Reducing dram image data access energy consumption in video processing
US8045021B2 (en) Memory organizational scheme and controller architecture for image and video processing
CN102547281A (en) Joint photographic experts group (JPEG) image compression concurrency control method based on digital signal processor (DSP)
Haidous et al. Content-adaptable ROI-aware video storage for power-quality scalable mobile streaming
CN105263021A (en) UVD-based HEVC video decoding method
CN102075759B (en) Low-power consumption encoding method for dynamic memory in video decoding application
CN201001175Y (en) IP audio/video coding decoder
CN104954798B (en) The straight-through display methods of video decoding and device
US20130278775A1 (en) Multiple Stream Processing for Video Analytics and Encoding
CN104168509B (en) Program editing method applicable to environment with various material sources
CN103596013B (en) novel decoder and decoding method
CN103369325B (en) A kind of low complex degree video compress transmission method being applicable to wireless video Sensor Network
CN115734004A (en) Video processing method, device, system and equipment
TWI565303B (en) Image processing system and image processing method
CN113055680B (en) Distributed transcoding method
CN103414898A (en) Method and system for collecting high-resolution video

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
PE01 Entry into force of the registration of the contract for pledge of patent right

Denomination of invention: Audio and video acquisition and play processing method with whole embedding of memory

Effective date of registration: 20171102

Granted publication date: 20141210

Pledgee: China Co truction Bank Corp Guangzhou economic and Technological Development Zone sub branch

Pledgor: Anyka (Guangzhou) Microelectronics Technology Co., Ltd.

Registration number: 2017990001008

PE01 Entry into force of the registration of the contract for pledge of patent right
PC01 Cancellation of the registration of the contract for pledge of patent right
PC01 Cancellation of the registration of the contract for pledge of patent right

Date of cancellation: 20181227

Granted publication date: 20141210

Pledgee: China Co truction Bank Corp Guangzhou economic and Technological Development Zone sub branch

Pledgor: Anyka (Guangzhou) Microelectronics Technology Co., Ltd.

Registration number: 2017990001008

PE01 Entry into force of the registration of the contract for pledge of patent right

Denomination of invention: Audio and video acquisition and play processing method with whole embedding of memory

Effective date of registration: 20190130

Granted publication date: 20141210

Pledgee: China Co truction Bank Corp Guangzhou economic and Technological Development Zone sub branch

Pledgor: Anyka (Guangzhou) Microelectronics Technology Co., Ltd.

Registration number: 2019440000051

PE01 Entry into force of the registration of the contract for pledge of patent right
PC01 Cancellation of the registration of the contract for pledge of patent right

Date of cancellation: 20200320

Granted publication date: 20141210

Pledgee: China Co truction Bank Corp Guangzhou economic and Technological Development Zone sub branch

Pledgor: ANYKA (GUANGZHOU) MICROELECTRONICS TECHNOLOGY Co.,Ltd.

Registration number: 2019440000051

PC01 Cancellation of the registration of the contract for pledge of patent right
CP01 Change in the name or title of a patent holder

Address after: 510663 3rd floor, area C1, innovation building, 182 science Avenue, Guangzhou Science City, Guangzhou City, Guangdong Province

Patentee after: Guangzhou Ankai Microelectronics Co.,Ltd.

Address before: 510663 3rd floor, area C1, innovation building, 182 science Avenue, Guangzhou Science City, Guangzhou City, Guangdong Province

Patentee before: ANYKA (GUANGZHOU) MICROELECTRONICS TECHNOLOGY Co.,Ltd.

CP01 Change in the name or title of a patent holder
CP02 Change in the address of a patent holder

Address after: 510555 No. 107 Bowen Road, Huangpu District, Guangzhou, Guangdong

Patentee after: Guangzhou Ankai Microelectronics Co., Ltd

Address before: 510663 3rd floor, area C1, innovation building, 182 science Avenue, Guangzhou Science City, Guangzhou City, Guangdong Province

Patentee before: Guangzhou Ankai Microelectronics Co., Ltd

CP02 Change in the address of a patent holder