CoreConverter and very large files

[fractaleater]

Using ref 12.1 on a machine with lots of muscle.

Got some whole album rips in wma lossless and converting them to flac is causing me some real trouble. I've converted tens of thousands of <= 200MB with no trouble. But around 200MB, something seems to go wrong. There's also some substantial memory allocation > 2GB while converting.

I'm looking into the issue, and I figured I would ask if anyone is aware of a problem with large files and coreconverter under tight physical memory conditions. The memory allocation does not seem abnormal as it grows size-proportional to the file you are converting, but I make a note of it since there's only 2GB of physical RAM in the machine. virtual limits out at around 6GB.

I notice that around 4GB of peak commit charge, the files just seem to stop converting, with no error notice. The progress bar does not get to 100%, and there are no tags in the "converted" files.

Richard

[LtData]

Re: CoreConverter and very large files

4GB is the limit that a 32-bit program can address. Your 200MB files, what format are they? If you look at the "Audio Properties" what size does it say the file is originally aka uncompressed?

[fractaleater]

Re: CoreConverter and very large files

Below is from the properties page.. when I try and convert this file using GUI interface, it says finished all of a sudden halfway through the conversion. At the point where it hits 2GB of commit charge. It leaves the incomplete file (204,111KB) on the hard drive and reports no error. Using a signed int as a ptr?

Artist: Dimitri from Paris
Title: Mixed Dance
Album: House Music All Night Long
Track: 1
Disc:
Genre: Dance
Year: 2005
Rating:
Composer:
Size: 530.59 MB (68% of original, 1.4 to 1 compression)
Original Size: 772.89 MB
Length: 1 hour 16 minutes
Channels: 2 (stereo)
Sample Rate: 44.1 KHz;
Sample Size: 16 bit
Bit Rate: 1,411 kbps
Encoder: Windows Media Audio 9.1 Lossless
Encoder Settings: VBR Quality 100, 44 kHz, 2 channel 16 bit 1-pass VBR
Audio Quality: Perfect (Lossless)
Play Count:
Last Played:
Contains: ID Tag [WMA ID Tag]
Channel Mapping:
File: Dimitri from Paris - House Mixed
Type: Windows Media Audio file [.wma]

[LtData]

Re: CoreConverter and very large files

What format are you trying to convert to? What settings are you using? Are you running any DSP effects?

[fractaleater]

Re: CoreConverter and very large files

Windows server 2003
Converting to FLAC
Compression Level 5
After encoding verify written audio [x]
Output location: Original Folder
DSP Effects: None

I upgraded from ref 12 to ref 12.1. I immediately attempted to do a conversion.

[Spoon]

Re: CoreConverter and very large files

>2GB of commit charge. It leaves the incomplete file (204,111KB)

Entirely down to windows, we do not handle allocations.

Try converting the file to test conversion, set at limited to x4 encode speed, it should take the true length / 4 if dbpoweramp can read the file.

[fractaleater]

Re: CoreConverter and very large files

On winserver2003 Using test decoder at 4X, it fails (but reports success) at 6min22sec of 17min process and around 2.1GB RAM used.

On XP (with 1GB RAM) using using test decoder at default, it succeeds in 2 min and around 1GB RAM used.

Both systems using ref12.1 and wma pro 9.

On the surface, seems to be an OS issue. But could also be a CPU issue since the XP machine uses P4 3.2GHz, and the 2k3 machine uses Xeon E5310 at 1.6GHz.

But the Xeon machine also uses something called "Physical Address Translation" as reported by "system". And has 2GB RAM. I guess that's a hardware virtual memory manager. Possibly something going wrong in there?

[Spoon]

Re: CoreConverter and very large files

If your virtual memory did not work then you would see windows crashing.

[fractaleater]

Re: CoreConverter and very large files

I used a boot.ini switch like: /MAXMEM=1024, and it still died at 2GB. A 32bit process can access 2GB of memory unless using the /3GB boot.ini switch. I tried the /3GB and /PAE switches, but I guess CoreConverter is not linked/compiled with /LARGEADDRESSAWARE. While windows 32bit total virtual address space (without /PAE switch) is 4GB our wall seems to be the 2GB per process barrier. It is still very strange though. Because XP converts the file without using so much memory.

XP uses around 1.1GB virtual memory to complete 100% of the 530MB wma file, and 2k3 uses 2.1GB of virtual memory to complete 1/3 of the same file. Something is going wrong someplace.

The E5310 is a Xeon chip, so I am considering redoing the OS install as 64bit. I have searched around, but there's not a definitive statement like: "ref 12.1 runs native 64bit on winxp 64bit, win server 2003 64bit." Is ref 12.1 compiled for 64 bit?

Richard

[LtData]

Re: CoreConverter and very large files

No, dMC is only a 32-bit program.

Can you convert your WMA lossless file to a WAV file?

[Spoon]

Re: CoreConverter and very large files

Our program will not be allocating that memory (we convert in small chunks), rather the wma subsystem built into the operating system, or some kind of file caching.

[fractaleater]

Re: CoreConverter and very large files

Not sure what it is. But I guess the test converter really is a test "decoder". So I went to Microsoft and downloaded their encoder so I could see what their "decoder" would do:

http://www.microsoft.com/windows/windowsmedia/forpros/encoder/default.mspx

This resulted in my 1 hour wma lossless track taking 8 min to convert to a wma 192 VBR (the default). The system used 180MB RAM total throughout the conversion. I guess that means something is going wrong even on XP where it is using 1GB RAM.

[Spoon]

Re: CoreConverter and very large files

Actually there was a buffer leak, it is now been fixed, it should take no more than 5 mb now...

[fractaleater]

Re: CoreConverter and very large files

I'm very happy to hear that it is fixed!

Is that fix included in the beta or in the 12.2 reference currently available through the update process?

[LtData]

Re: CoreConverter and very large files

The WMA codec with the fix has been released, use the "Check for Update" button in dBpoweramp Configuration to get it.