PDA

View Full Version : CoreConverter and very large files



fractaleater
06-07-2007, 12:58 PM
Using ref 12.1 on a machine with lots of muscle.

Got some whole album rips in wma lossless and converting them to flac is causing me some real trouble. I've converted tens of thousands of <= 200MB with no trouble. But around 200MB, something seems to go wrong. There's also some substantial memory allocation > 2GB while converting.

I'm looking into the issue, and I figured I would ask if anyone is aware of a problem with large files and coreconverter under tight physical memory conditions. The memory allocation does not seem abnormal as it grows size-proportional to the file you are converting, but I make a note of it since there's only 2GB of physical RAM in the machine. virtual limits out at around 6GB.

I notice that around 4GB of peak commit charge, the files just seem to stop converting, with no error notice. The progress bar does not get to 100%, and there are no tags in the "converted" files.

Richard

LtData
06-07-2007, 01:21 PM
4GB is the limit that a 32-bit program can address. Your 200MB files, what format are they? If you look at the "Audio Properties" what size does it say the file is originally aka uncompressed?

fractaleater
06-07-2007, 03:23 PM
Below is from the properties page.. when I try and convert this file using GUI interface, it says finished all of a sudden halfway through the conversion. At the point where it hits 2GB of commit charge. It leaves the incomplete file (204,111KB) on the hard drive and reports no error. Using a signed int as a ptr?

Artist: Dimitri from Paris
Title: Mixed Dance
Album: House Music All Night Long
Track: 1
Disc:
Genre: Dance
Year: 2005
Rating:
Composer:
Size: 530.59 MB (68% of original, 1.4 to 1 compression)
Original Size: 772.89 MB
Length: 1 hour 16 minutes
Channels: 2 (stereo)
Sample Rate: 44.1 KHz;
Sample Size: 16 bit
Bit Rate: 1,411 kbps
Encoder: Windows Media Audio 9.1 Lossless
Encoder Settings: VBR Quality 100, 44 kHz, 2 channel 16 bit 1-pass VBR
Audio Quality: Perfect (Lossless)
Play Count:
Last Played:
Contains: ID Tag [WMA ID Tag]
Channel Mapping:
File: Dimitri from Paris - House Mixed
Type: Windows Media Audio file [.wma]

LtData
06-07-2007, 05:01 PM
What format are you trying to convert to? What settings are you using? Are you running any DSP effects?

fractaleater
06-08-2007, 03:43 AM
Windows server 2003
Converting to FLAC
Compression Level 5
After encoding verify written audio [x]
Output location: Original Folder
DSP Effects: None

I upgraded from ref 12 to ref 12.1. I immediately attempted to do a conversion.

Spoon
06-08-2007, 04:13 AM
>2GB of commit charge. It leaves the incomplete file (204,111KB)

Entirely down to windows, we do not handle allocations.

Try converting the file to test conversion, set at limited to x4 encode speed, it should take the true length / 4 if dbpoweramp can read the file.

fractaleater
06-09-2007, 05:18 AM
On winserver2003 Using test decoder at 4X, it fails (but reports success) at 6min22sec of 17min process and around 2.1GB RAM used.

On XP (with 1GB RAM) using using test decoder at default, it succeeds in 2 min and around 1GB RAM used.

Both systems using ref12.1 and wma pro 9.

On the surface, seems to be an OS issue. But could also be a CPU issue since the XP machine uses P4 3.2GHz, and the 2k3 machine uses Xeon E5310 at 1.6GHz.

But the Xeon machine also uses something called "Physical Address Translation" as reported by "system". And has 2GB RAM. I guess that's a hardware virtual memory manager. Possibly something going wrong in there?

Spoon
06-09-2007, 10:06 AM
If your virtual memory did not work then you would see windows crashing.

fractaleater
06-11-2007, 07:31 PM
I used a boot.ini switch like: /MAXMEM=1024, and it still died at 2GB. A 32bit process can access 2GB of memory unless using the /3GB boot.ini switch. I tried the /3GB and /PAE switches, but I guess CoreConverter is not linked/compiled with /LARGEADDRESSAWARE. While windows 32bit total virtual address space (without /PAE switch) is 4GB our wall seems to be the 2GB per process barrier. It is still very strange though. Because XP converts the file without using so much memory.

XP uses around 1.1GB virtual memory to complete 100% of the 530MB wma file, and 2k3 uses 2.1GB of virtual memory to complete 1/3 of the same file. Something is going wrong someplace.

The E5310 is a Xeon chip, so I am considering redoing the OS install as 64bit. I have searched around, but there's not a definitive statement like: "ref 12.1 runs native 64bit on winxp 64bit, win server 2003 64bit." Is ref 12.1 compiled for 64 bit?

Richard

LtData
06-11-2007, 08:45 PM
No, dMC is only a 32-bit program.

Can you convert your WMA lossless file to a WAV file?

Spoon
06-12-2007, 03:48 AM
Our program will not be allocating that memory (we convert in small chunks), rather the wma subsystem built into the operating system, or some kind of file caching.

fractaleater
06-21-2007, 06:49 AM
Not sure what it is. But I guess the test converter really is a test "decoder". So I went to Microsoft and downloaded their encoder so I could see what their "decoder" would do:

http://www.microsoft.com/windows/windowsmedia/forpros/encoder/default.mspx

This resulted in my 1 hour wma lossless track taking 8 min to convert to a wma 192 VBR (the default). The system used 180MB RAM total throughout the conversion. I guess that means something is going wrong even on XP where it is using 1GB RAM.

Spoon
07-02-2007, 06:49 AM
Actually there was a buffer leak, it is now been fixed, it should take no more than 5 mb now...

fractaleater
07-03-2007, 07:18 AM
I'm very happy to hear that it is fixed!

Is that fix included in the beta or in the 12.2 reference currently available through the update process?

LtData
07-03-2007, 08:27 AM
The WMA codec with the fix has been released, use the "Check for Update" button in dBpoweramp Configuration to get it.

fractaleater
07-08-2007, 08:41 AM
Update check does not report that there is a new version of wma encoder.

I guess only the wma 10+ codec is fixed? I am running on Wins2K3, so I believe I am restricted to version 9 of the wma codec. Is there a fix for the version 9 of the codec? I have release 1 of wma pro 9 encoder installed, and a generic wma read decoder.

Is version 10+ of the wma encoder also for winserver2003?

Spoon
07-08-2007, 09:15 AM
Should be for 2003 as well yes, it is newer than xp, install the xp version.

fractaleater
07-10-2007, 04:33 AM
Yup.. installing codec wma 10 worked just fine. CoreConverter converts my huge (500MB+) wma file with no trouble to any format. It uses a tiny amount of memory (less than 20MB) to process.

Thanks very much for the attention to this matter!

Spoon
07-10-2007, 08:12 AM
Thanks for bringing it to our attention!