Secure ripping using VirtualBox, ECC, ZFS and open questions

[dpsig]

Hi all,

I have read all related posts to people which tried a similar setup. First to give you an impression what it is about, I am currently setting up a system for (finally the goal) batch ripping of my CDs (roughly 800-1000). Goal: Make "as sure as possible" to have a bit-correct and safe rip of the originals which also stays correct over time. With "as sure as possible" I refer to decisions for the overall system layout which will at least incorporate freely available and affordable technologies. We can have a long discussion about the pros and cons (and necessity) but here I want to first describe the

main requirements:

R1) Use ZFS mirrored file system for storage
R2) Use ECC RAM on machine(s) for the rip and the storage
R3) Use dbPowerAmp's secure ripping features
R4) Use CD/DVD drives from the top of the CD/DVD Drive Accuracy List 2016:

https://forum.dbpoweramp.com/showthread.php?37706-CD-DVD-Drive-Accuracy-List-2016

To fulfill R1 and R2 with just one machine I decided to give Virtualization based on VirtualBox a try. The only two alternatives fulfilling the requirements would have been to a) use separate machines for ripping and storing or b) use OSX for both (dbPA is available for OSX as is ZFS). a) Surely is more costly and b) includes a lot of potential pitfalls like installing OSX on non Apple hardware (and finding supported hardware in the first place) and install (not officially supported) ZFS there (with all options). The current system is using

Hardware:

- Supermicro MBD-X10SLM+-F 32GB ECC
- Intel Xeon E3-1240LV3 4x 2.0GH
- HD: 2x WD100EFAX red 3,5" 10TB SATA (data)
- HD: 1x Intel DC S3700 SSD 2,5" 100GB (system)
- CD/DVD: 10x LITE-ON HAS124-14 (F)
- CD/DVD: 1x LG GH22NP20 ATAPI (alternative for "unwilling" CDs)
- CD/DVD: 1x NEC ND-1300A ATAPI (alternative for "unwilling" CDs)
- 10x Renkforce USB 2.0 to IDE+SATA (CD/DVD drive attachement)
- 2x Renkforce USB 3.0 to IDE+SATA (CD/DVD drive attachement)

Software:

- Host OS: Ubuntu 16.04 LTS
- VirtualBox V 5.1.24
- Client OS: Windows 10 Enterprise 64bit (latest release and security packs)
- dbPowerAmp R16.2 Windows 64

I spare you the overall installation details/nightmare, at the end I am having a running Windows 10 client on an headless Ubuntu host with all CD/DVD drives assigned to the client and accessible from within dbPA. As is the ZFS HD pool for the data. Hence, current goal is to setup the secure ripping as described here:

https://www.dbpoweramp.com/cd-ripper-setup-guide.htm

As has been posted here before, e.g. in

https://forum.dbpoweramp.com/showthread.php?24593-AccurateRip-key-discs-and-VirtualBox&highlight=virtualbox

the DVD drives are only listed in the client OS as "VBOX - CD-ROM" hence

a) I had to exclude my two non iHAS backup drives and only use the 10 identicial drives for now and
b) I need manual detection of properties:

P1) I could manage to successfully configure AccurateRip for the 10 identical drives with the following CDs from my collection:

- Phil Collins, Face Value
- Heaven 17, The Luxury Gap
- Annie Lennox, Diva

P2) The offset was detected as +6 samples, +24 bytes.
P3) Also, the cache detection returned and reported "no cache"
P4) Detecting C2 errors with the black triangle does not work. Detection never returns and sometimes dbPA crashes and the current user is automatically logged out of windows.

In VirtualBox the drives are configured as "passthrough", otherwise the Audio CDs are not even shown. They are attached to the (virtual) SATA adapter.

Hence P3 and P4 are giving me headaches currently and are defining my questions to some of you can hopefully answer:

- I am not sure if there are really drives without cache nowadays. Maybe some owners of the iHAS 124 can comment on this? Maybe it is a VirtualBox thing too. Is it safe to leave it non-cached?

- P4 is a show stopper now. Is it better (or different) to use the black triangle test (as described here https://www.dbpoweramp.com/cd-ripper-setup-guide.htm) or to use a heavily scratched CD as guided to in the online help of dbPA?

- I know that many posts always refer to "let dbPA do the detection of drive properties" but in the virtualization case it seems at least favorable to define these properties (e.g., cache and C2) by hand since the drive type is occluded by the virtualization. Would that be safe to set properties here manually even though the automatic detection reported a different outcome (e.g., no cache) or failed (e.g., C2 detection)?

- Would owners of Liteon iHAS 124 F be so kind as to report their setting for the drive they are using?

Thanks for your help

[Spoon]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

Running a virtualized os for ripping is asking for trouble, and gives no advantages of having ZFS or ECC, because if you rip to something like FLAC then the source audio CRC is stored in the file so it can be checked at any time by converting the FLAC file to 'Test Conversion', if there were any issues it would be shown.

[dpsig]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

Running a virtualized os for ripping is asking for trouble, and gives no advantages of having ZFS or ECC, because if you rip to something like FLAC then the source audio CRC is stored in the file so it can be checked at any time by converting the FLAC file to 'Test Conversion', if there were any issues it would be shown.

Dear Spoon,

Thanks for your reply. As for the first part (virtualized ripping and asking for trouble) I would slightly agree taking into account the problems I encountered so far. But yours is the first answer where I actually read a definite "not advised". There had been others posting similar approaches here but the overall idea never was discouraged, was it? It would actually nice to get some replies to the open questions posted. I tested a few rips so far but all were verified by AccurateRip, hence no need for C2 and alike. I just want my system to be prepared for the first CDs which will fail on AccurateRip and will need additional secure rip measures.

As for the second part I would strongly disagree, though. Anything you do with CRC calculation and testing could potentially be hampered by false bits in RAM. But this is not the main rational since it could be detected during rip time and post checks as you wrote. The main rational for ECC is running ZFS without ECC is just a really bad idea. Major goal is consistency and potentially non ECC RAM contents could be defective and could be written as defective to the file system without any chance to know about the problem. I guess it is not completely wrong to follow well established practices from data centers here.

The real problem is, being able to "check the CRC at any time" as you mentioned is no real help. One would have to do this manually but when? Only at rip time? Not enough. One would not be aware of any changes (e.g., bit rot) on the file system which could occur over time (and which are not so seldom with todays >10TB HDs). A scheduled scrub of the ZFS file system would automatically flag integrity problems and a mirrored file system can automatically repair these problems. What do I gain if I want my rips as perfect copies of my CDs (to also save the content from aging problems), later I detect a mismatch in the CRC but can't restore the original? It is not enough to be potentially able to detect errors but a) its up to you to do the test and b) if you can't repair them in case. My goal is to follow your advice "Rip once but rip right" trying to really save the CDs' content over time.

best

[dpsig]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

Unfortunately I haven't got any specific answers to my CD drive-related questions from my initial post hence I started a new thread without all the distracting information about the overall system. Don't know why this is not approved by the forum admins but the new post is not showing up for days now. Hence, if anyone has information about the SecureRip setup for the LITEON iHAS 124-14 (aka F version), that would be really appreciated. Only got a few days trial remaining and still don't know if buying a license would be a good idea.

[mville]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

The system you describe is overkill, IMO.

A Windows 10 PC, a decent optical drive (check these forums for recommendations), dBpoweramp Music Converter, the flac file format, local and cloud backup (and possibly off-site backup), is really all you need.

I have been building my audio library for 20 years and have yet to have a corrupted flac.

[mville]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

if anyone has information about the SecureRip setup for the LITEON iHAS 124-14 (aka F version), that would be really appreciated.

Try reading the online help e.g. dBpoweramp CD Ripper

[mville]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

P4) Detecting C2 errors with the black triangle does not work. Detection never returns and sometimes dbPA crashes and the current user is automatically logged out of windows.

Drive firmware often reports C2 support incorrectly. If you have problems with the C2 Error Pointers for Error Detection option, turn it off. What you need to be testing for is whether you can get AccurateRips, or not. If you get successful AccurateRips, your drive is configured OK.

[mville]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

The real problem is, being able to "check the CRC at any time" as you mentioned is no real help. One would have to do this manually but when? Only at rip time? Not enough.

I run Test Conversion on my audio library periodically, to check the health of my flacs. It is enough and is no big deal.

[dpsig]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

Try reading the online help e.g. dBpoweramp CD Ripper

Sorry to say, in my initial post above I clearly cited that I am exactly trying to follow the online help you are referring to. In fact, the help page abut setting-up secure ripping contradicts the process guided by the online help. The first wants to use the black triangle disk, the latter wants to use a really scratched disk. So its all the same?

[dpsig]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

The system you describe is overkill, IMO.

It is overkill to want to use 10 CD drives simultaneously? Isn't this where batch ripper is made for? The rest of the system is only very picky in terms of data integrity and is the main data storage for the household anyways.

A Windows 10 PC, a decent optical drive (check these forums for recommendations), dBpoweramp Music Converter, the flac file format, local and cloud backup (and possibly off-site backup), is really all you need.

That is exactly what I am doing, see my specs. I am using the top of the recommended drives from the 2016 list and was only asking for people with the same drive to confirm my current results and to report on their C2 and cache setups since the latter two did not successfully finish in the virtualized setup.

I have been building my audio library for 20 years and have yet to have a corrupted flac.

Ok, I don't want to get picky here but I would doubt you would be aware of a corrupted flac unless you manually check the CRC included for every track or listen to every track in you (large i guess) library repeatedly as you confirmed on doing also. I just want to have this manual check been made by the file system to cover all of my (also backup) data.

But I really didn't want to start a discussion about data integrity and alike since there are obviously many different opinions about it. I just asked for some advice for setting up the secure ripping since this did not work for me (following the online help and manuals).

[dpsig]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

Thanks for the reply. AccurateRip does work so far for my setup. But I do expect some disks in my 1000+ collection with some disks > 30 years showing up with errors, would I? For these ones I'd like a setup to really rip as secure as possible using all measures provided (e.g., n-times overread with slower speeds, C2 if possible).

I thought that all the option to secure rip besides AccurateRip are really the distinctive features of dbPA. Otherwise one could just use one of the other available rip tools which also support AccurateRip if not for these features.

[Spoon]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

Commercial ripping houses use a maximum of 4 drives per system ( and multiple systems ), putting 10 drives on Windows is not reliable, regardless of the software used (it leads to multiple low level stalls.

[dpsig]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

Commercial ripping houses use a maximum of 4 drives per system ( and multiple systems ), putting 10 drives on Windows is not reliable, regardless of the software used (it leads to multiple low level stalls.

Oh, would have been nice to know that in advance before buying the hardware. Honestly, I thought this setup is the target for batch ripper and I also thought I have read here in the forum that people were using multiple drives, e.g.:

https://forum.dbpoweramp.com/showthread.php?39934-Ripping-multiple-CDs-at-once-in-parallel-so-to-speak-occasional-cross-contamination&highlight=multiple+drives

https://forum.dbpoweramp.com/showthread.php?38980-Is-there-a-guide-for-using-multiple-drives-with-a-single-PC-to-simultaneously-rip-Cds&highlight=multiple+drives

But more digging turned out several threads with similar problems. So I think there are still two options:

1. Using OSX then as rip system. Does the same limit apply to this OS also (i have all OSX versions since beginning if version is a problem)?

2. Maybe after all the idea to run this on VirtualBox is not that stupid after all, just have to clone the initial Windows system 3x and assign 4 drives to each system. I know you don't like this idea but I might give it a try (but would prefer the OSX solution). Is it really a Windows problem or a problem of the underlying controller hardware?

best

[mville]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

Ok, I don't want to get picky here but I would doubt you would be aware of a corrupted flac unless you manually check the CRC included for every track or listen to every track in you (large i guess) library repeatedly as you confirmed on doing also. I just want to have this manual check been made by the file system to cover all of my (also backup) data.

As I said earlier, I run the Test Conversion. Are you saying that Test Conversion does NOT report corrupted flacs?

... also, if flacs were corrupt, then this would be reflected in the PerfectTUNES AccurateRip module, which I also run.

I rely on these tools to help me in the area of file corruption. If you doubt that these tools will report corruption, as I had assumed, I would like to know. Please can you enlighten me and by all means, be picky as this is important stuff?

[garym]

Re: Secure ripping using VirtualBox, ECC, ZFS and open questions

As I said earlier, I run the Test Conversion. Are you saying that Test Conversion does NOT report corrupted flacs?

And test conversion is not really a manual process (it's a batch process). I use it as well every so often to confirm the integrity of various HDD backups of my ~ 100,000 track library. A couple of mouse clicks and a few hours later I have confirmation that all the flac files are NOT corrupted. If one was, it would identify that as well. Very useful tool.