dBpoweramp Batch Ripper: Discussions

[bhoar]

Re: dBpoweramp Batch Ripper: Discussions

RTW -

Personally I think the best cost trade-off on systems is reached at a 6 drive system. I have an overclocked Quad core system that can just about keep up with 5/6 encoding streams. If you go above 6 drives then you need to start looking at more expensive cases (if your using internals) and things can start getting pretty messy. The hard drive will also start to get maxed out at some where over 6 drives (I did the calculation previously, but can't find them), then you are into RAID systems to keep up with the data, and then more memory to keep up with the extra drives etc.

The back of the envelope calculations I did were based on 40x extraction (4x150kBps=6MBps per optical drive). Six drives is 36MBps WAV extraction, 23.4MBps lossless (~65%), and substantially less for lossy. Contemporary drives can stream higher than that, but since it is one write thread per drive, you probably wouldn't want to push it.

On the flip side, I plan to be writing to a RAID-0 array, which should double the potential streaming performance. I'll have to test to see if that improves anything with six or more write streams.

If I were trying to do a >6 drive (manual load) system on a budget and were building from scratch, I'd probably spend the $150 on the Cooler Master Stacker (aka "CM Stacker") case with the 10 external 5.25 bays. That's not too expensive, considering eight bay external enclosures run at least $115 or so, which would be the option if you already have the computer constructed or prefer a smaller on-desktop footprint.

And finally: for manual load systems, I think it's probably OK to have more optical drives than your system can handle (assuming there aren't any severe performance penalties): even if the system can't keep up with the drives/encoding, you have reduced the number of manual unload/load cycles the operator has to perform.

I also wouldn't personally touch anything involving IDE. I used to have a system that used IDE to SCSI converters which ran OK, but each drive ran with its own controller before plugging into the SCSI adapter. Even using 2 drives on the same controller can really cause problems if you hit any errors with a disc. If you are doing this to use a particular model of IDE drive then fair enough, but the far more robust solution is to use SATA drives.

IDE controllers weren't designed to handle that kind of data throughput (and certainly not in both directions at the same time), SATA ones were. Most new motherboards also come with PCIe slots so adding more SATA controllers is easy if you don't want to overload the onboard one. Extremely easy to build a system with SATA and it is lightening fast.

Agreed about using optical drives on IDE. Microsoft's XP IDE driver behavior is just plain idiotic when it comes to dealing with scratched discs in drives on IDE channels. Spoon has said that Vista's IDE driver appears to be a lot less problematic in this area. I'm using Server 2003 EE*, which I assume probably has similar issues to the XP driver.

I am using Firewire for all optical drive connectivity, which is essentially the same approach you took, except with a higher speed bus (firewire being, essentially, a pre-SAS serial based SCSI connection).

Regarding SATA optical drives, two things:

1. From what I've read a lot of them on the market are essentially ATAPI drives with an onboard SATA<->ATAPI bridge chip. Since I already have plenty of ATAPI optical drives, I'm not going to be buying any new ones until these wear out. I might do some experimenting with the $6 SATA<->IDE bridge modules to see how they fare against the Firewire<->IDE bridges, which cost a lot more.

2. SATA can run in "Native" and "Compatible" modes. IIRC, one would want to run in "Native" mode if possible, since "Compatible" mode mimics an IDE connection, windows uses the standard IDE driver for the port and so Windows will end up doing the same brain-dead move and set the connection to PIO mode after a lot of failed reads. Is that right?

You can get into the debate about the accuracy of the drives etc. But I haven't started offering extremely paranoid rips to my customers yet anyway so its not an issue at present. Happy to test my current setup for accuracy etc. if others are interested.

I'll be needing to do that as well.

-brendan

* the box is performing double duty (ripping, vmware)

[Spoon]

Re: dBpoweramp Batch Ripper: Discussions

>I think it's probably OK to have more optical drives than your system can
>handle (assuming there aren't any severe performance penalties):

I would not, depends on the - take my Plextor 230a, very accurate drive, unless the system is running 6 drives and the 230a is not being allowed to rip at full speed, the results were not accurate (on a clean cd).

[bhoar]

Re: dBpoweramp Batch Ripper: Discussions

>I think it's probably OK to have more optical drives than your system can
>handle (assuming there aren't any severe performance penalties):

I would not, depends on the - take my Plextor 230a, very accurate drive, unless the system is running 6 drives and the 230a is not being allowed to rip at full speed, the results were not accurate (on a clean cd).

Let me clarify: I meant, in this case, that it would be acceptable to have some overload in the encoding part of the pipeline (not the ripping part). Anything that impacts the optical drive IO is probably going to cause problems because of non-optimal disc read patterns.

Hence my earlier question about the CDGrab.exe's encoder priority settings.

As long as there isn't any overload in the reading IO channels (be they SATA or firewire), and there's at least enough CPU power left while encoding to manage the read strategy dispatching (by setting encoding to lower priority), the sort of problem Spoon mentioned might be avoided...right?

Hmm, I suppose I am making some assumptions about how far encoding is allowed to lag behind the ripper.

Spoon - regardless if it turns out I am hopeless wrong (or not), how about another feature, very easy to implement and a boon to non-automated rippers everywhere: a user-configurable feature to clamp the maximum number of drives allowed to be actively ripping at once, without having the operator disable drives from use in the batch session?

The easy implementation would be to put all ready drives above a certain number in a wait queue. e.g. at setting of 4, even if you have eight drives active and loaded, only four would be ripping at once. This allows end users to have the benefit of eight discs loaded per unload/load cycle without the side effects you saw.

A perhaps more complicated implementation would be to do this at the track level, by limiting the number of CDGrab.exes and putting the pending calls in a wait queue.

-brendan

[bhoar]

Re: dBpoweramp Batch Ripper: Discussions

In my last four-drive test, using the Test Conversion codec (no encoding) on several ~75-85% full CDs, the max combined speed I saw, right near the end of the CDs, was 149x. It was generally > 130 during the last 1/3 of the run, until one of the discs was finished, which immediately drops down the total.

Hit a high-water mark of 172x earlier tonight with FLAC at standard using six drives on a four core system. Still seeing only four drives able to rip at full speed at once, the last two drives are at 1/4 to 1/2 speed. I've tried swapping out firewire<->ATAPI bridges and/or firewire controllers but get the same result. I even started playing with Microsoft's Interrupt Affinity Tool (and filter driver) to see if I could somehow move all the PCI IRQs for firewire to one or two CPUs to see if that made a difference and it did not.

Planning to try SATA<->ATAPI bridges next week.

I'm wondering if the drives are resetting their speed to non maximum, if CDGrab.exe is setting their speed to non maximum, or...?

None of the cores are seeing anywhere near to full utilization.

-brendan

[Spoon]

Re: dBpoweramp Batch Ripper: Discussions

It will be a DMA limit I am guessing, perhaps try:

some drives USB
some drive Firewire
some IDE
some SATA

then you would get maximum speed.

[bhoar]

Re: dBpoweramp Batch Ripper: Discussions

It will be a DMA limit I am guessing, perhaps try:

some drives USB
some drive Firewire
some IDE
some SATA

then you would get maximum speed.

Tried the experiment with a 4 firewire/2 usb 2.0 split but that just made things worse (note, it appears that the firewire chipset and the Enhanced USB aka 2.0 are both on APCI IRQ 23, which might be one reason why). When the SATA->ATAPI bridges arrive, I'll try a 3-4 firewire/2-3 SATA split.

The weirdest part of the experiments so far is that if I use the Test Conversion codec instead of the FLAC codec, the measured speed never gets nearly as high. It's almost as if it needs the occasional spikes of CPU use, the large buffer copies and/or the large hard disk writes in order to allow the optical drives to extract data at the highest speed.

Perhaps I should be keeping records of the results.

-brendan

[bhoar]

Re: dBpoweramp Batch Ripper: Discussions

Some requests/suggestions:

- In the Results tab of the Batch Ripper, can the rip speed and/or rip time be listed for each disc?

- Related to the above: if more space is needed in the three lines next to each disc in the results, perhaps the disc # shown could be changed to a larger point size in a 25-30% grey background behind the text (starting at 001 or 0001).

- Can you add a pause/resume batch capability? e.g. if one needs to add more discs or clear rejects manually, it might be nice to know there are no pending robotic requests set to cause the picker to intersect with your hand. A save/resume capability might also be nice, if there's a need to reboot during a long batch.

-brendan

[bhoar]

Batch Ripper throughput issues. (was Re: dBpoweramp Batch Ripper: Discussions)

---Batch Ripper throughput issues.---

I'm testing with burst mode for max throughput tests. Machine is an Intel Core 2 Q6600 quad board with on board Firewire (1 external/2 internal).

On each drive, Ripping seems to randomly start at either a normal 12x to 20x or sometimes 7x. If the former, it increases toward the 40x end of the spectrum as it rips closer to the end of the disc. If the latter, it stays at 7x-10x for most of the disc.

This is regardless of whether I use the FLAC encoder (with CPU load) or the Test encoder (with no CPU load)

The symptom Moves from drive to drive, and is not associated with which discs (I'm testing with clean/good condition audio CDs at this point). It seems to be more prevelent when I have more than two drives running (up to six) in parallel. But it can still happen with just two drives.

I started with 6 firewire drives on three firewire bridges with master/slave support on a single Firewire 400 chain. Thinking that perhaps the issue was hardware limitations in terms of IOPS or bandwidth, I've tried:

1. Moving from a single firewire chain to one bridge per port (3 ports), which involved adding two 7-pin to external firewire cable/bracket combos to move the internal motherboard firewire ports to the rear of the machine.

No change.

2. Tried a couple of different types of firewire to IDE bridge chipsets.

No change.

3. Using a PCI Firewire card to expand the # of firewire ports, similar to the above.

No change.

4. Tried moving a couple drives to USB to IDE bridge chipsets.

Got worse, I blame USB. Moved back to firewire.

5. Using the Interrupt Filter Configuration Tool from Windows XP's/Server-2003's Powertools to assign all firewire interrupts to a single CPU instead of allowing them to be served round-robin by one of the four cores. Typically this is used for servers moving a lot of data via gigabit ethernet over several network cards at once to take advantage of CPU caching patterns. (note: this seemed to have no impact on the # of interrupts per second per CPU reported via the microsoft performance tools.)

No change.

6. Tried putting two or three of the drives on SATA<->IDE bridges instead of firewire, using the motherboard SATA ports.

No change.

7. As above, but changing BIOS settings and installing Intel's storage matrix drivers so that the SATA ports are no longer in IDE emulation mode, but rather Native/RAID or Native/AHCI mode.

No change.

8. Monkeying a bit with the Firewire controllers PCI Latency settings (it's the only device on the motherboard with a non-zero value of 32). Tried 16 and 64.

No change.


---

Ideas, anyone?

Also, sinserting a disc in another drive can force a drive running at the former speed to revert to the slower speed. This seems unrelated to the relationship between the drives/ location of the drive on any particular bus.

-brendan

[Spoon]

Re: Batch Ripper throughput issues. (was Re: dBpoweramp Batch Ripper: Discussions)

Many cd drives, when you insert a cd will read at the end of the disc, and decide the speed to use for the whole cd, plextor drives are like this.

[bhoar]

Re: Batch Ripper throughput issues. (was Re: dBpoweramp Batch Ripper: Discussions)

Many cd drives, when you insert a cd will read at the end of the disc, and decide the speed to use for the whole cd, plextor drives are like this.

As I noted in above, repeating experiments, the result seem to be unrelated to the disc. The same CD would rip at different speeds in the same drive in different batch sessions. Plus, starting a rip in another drive would often cause a 28x rip in an existing drive to suddenly fall down to 7x immediately (in the middle of the disc). A sudden fall in disc read speed is strongly correllated with starting up additional IO.

It seems to point to some sort of windows IO issue to me. It seemed sort of random ripping from one drive always seemed to give the top speed. Ripping from two drives almost always gave the top speed. Ripping from three was about 66% correlated with getting the top speed from all three. Ripping from four drives was ~33% correlated with getting the top speed from all four. Above that, one or more drives *always* was ripping at low speed.

-brendan

[CiXel]

Re: dBpoweramp Batch Ripper: Discussions

Where are individual batch profiles stored so they can be moved between users or computers?

[Spoon]

Re: dBpoweramp Batch Ripper: Discussions

Registry >> HKCU >> Software >> Illustrate

[CiXel]

Re: dBpoweramp Batch Ripper: Discussions

Registry >> HKCU >> Software >> Illustrate

Great. Thanks for that.

While I can do it via the registry manually, Might it be beneficial to setup an import/export option for this where it can write the values to a file and then can be sucked into a new instance?

[Spoon]

Re: dBpoweramp Batch Ripper: Discussions

Eventually we will offer a network mode for batch ripper, allowing multiple computers to be controlled from one work station, ie you have x drives on computer 1, x drives on computer 2, x drives on computer 3, x drives computer 4. On one of the systems would be the master controller which combines all into one screen (ok multi windows for multi monitor system).

[CiXel]

Re: dBpoweramp Batch Ripper: Discussions

Ah, That would make sense and would make for a very powerful system.

A few observations after working with the current state today-
On the batch front
I would like to see a more verbose description of what's going on during a batch rip. Perhaps something akin to the "rip status" dialog in the single ripper so I can see when it's re-ripping frames or if it's stuck somewhere. (I got stuck on a few discs today. One was just taking an extra long time and eventually progressed, but on others it just sat there for hours during frame re-rips. I just wasn't sure what was going on and figured out later while doing a manual cd rip that it had stalled during that stage, as it did the same thing.) To that end, perhaps an option to skip a track if the elapsed time exceeds X times track length. (10x might be a good default) On once particular track, it spent 2 hours there stalled on a 5 minute song. I happened to catch it, but otherwise it would have sat there and nothing else would have gotten done.

It would also be great if we could drag and drop a cover art into say the results window to have it applied with our defined album art options.


In the regular CD ripper window
I can't seem to find and easy way to manually edit the 'artist' field as I can for the title field (ie. I can't click on artist and modify it). As it stand snow, I swap the title and artist, input the artist info, and then swap back. Am I just missing something here?
(also the ability to drag and drop in cover art within here too would be good)

Lastly, while utilizing a Sony changer...
Since the Sony changers maintain their disc order, and the programs remembers the "disc #" It would be great to be able to "goto" a disc number in order to rerip a file or manually enter metadata for those discs missing it.

Also a minor point, I've noticed if there is a disc mounted in the Sony changer slot when a batch rip begins, dB will start ripping that mounted disc first, and then will proceed with the batch rip. On the bright side, the implementation of a reject based on 'CD Same a last' works as intended, but if there was a way to go straight to the batch ripping it might clean things up.