CUETools DB Support

[EliC]

This is a feature that would REALLY help make the next version of dBpoweramp a true step forward!

http://db.cuetools.net/

What's it for?
You probably heard about AccurateRip, a wonderfull database of CD rip checksums, which helps you make sure your CD rip is an exact copy of original CD. What it can tell you is how many other people got the same data when copying this CD. CUETools Database is an extension of this idea.
What are the advantages?

* The most important feature is the ability not only to detect, but also correct small amounts of errors that occured in the ripping process.
* It's free of the offset problems. You don't even need to set up offset correction for your CD drive to be able to verify and what's more important, submit rips to the database. Different pressings of the same CD are treated as the same disc by the database, it doesn't care.
* Verification results are easier to deal with. There are exactly three possible outcomes: rip is correct, rip contains correctable errors, rip is unknown (or contains errors beyond repair).
* If there's a match, you can be certain it's really a match, because in addition to recovery record database uses a well-known CRC32 checksum of the whole CD image (except for 10*588 offset samples in the first and last seconds of the disc). This checksum is used as a rip ID in CTDB.

What are the downsides and limitations?

* CUETools DB doesn't bother with tracks. Your rip as a whole is either good/correctable, or it isn't. If one of the tracks is damaged beyound repair, CTDB cannot tell which one.
* If your rip contains errors, verification/correction process will involve downloading about 200kb of data, which is much more than it takes for AccurateRp.
* Verification process is slower than with AR.
* Database was just born and at the moment contains much less CDs than AR.

How many errors can a rip contain and still be repairable?

* That depends. The best case scenario is when there's one continuous damaged area up to 30-40 sectors (about half a second) long.
* The worst case scenario is 4 non-continuous damaged sectors in (very) unlucky positions.

What information does the database contain per each submission?

* CD TOC (Table Of Contents), i.e. length of every track.
* Offset-finding checksum, i.e. small (16 byte) recovery record for a set of samples throughout the CD, which allows to detect the offset difference between the rip in database and your rip, even if your rip contains some errors.
* CRC32 of the whole disc (except for some leadin/leadout samples).
* Submission date, artist, title.
* 180kb recovery record, which is stored separately and accessed only when verifying a broken rip or repairing it.

Spoon, if you don't want to add support, at least PLEASE open up the DSP architecture so that support could be added.

[EliC]

Re: CUETools DB Support

Here is a link to a CTDB thread at HA:

http://www.hydrogenaudio.org/forums/index.php?showtopic=79882

[Cad]

Re: CUETools DB Support

I was reasonably interested in CTDB until I saw this in your post:

"CUETools DB doesn't bother with tracks. Your rip as a whole is either good/correctable, or it isn't. If one of the tracks is damaged beyound repair, CTDB cannot tell which one."

Many (most?) of my rips are just one (or a few) tracks from compilation CDs, because they are tracks I don't already have, but the rest of the tracks on the CD are ones I do already have.

I don't like the idea of being forced to rip an entire CD just to 'recover' the one or two tracks I want, especially when dbPowerAmp already does such a spiffing job of being able to correctly rip damaged CDs, even if they are not in AccurateRip.

Also, does this CTDB thing take account of the fact that US and European releases of CDs sometimes have different tracks, or are mastered such that the track offsets on the US/European/Asian pressings are subtly different?
--
BFN
CAD

[EliC]

Re: CUETools DB Support

I was reasonably interested in CTDB until I saw this in your post:

"CUETools DB doesn't bother with tracks. Your rip as a whole is either good/correctable, or it isn't. If one of the tracks is damaged beyound repair, CTDB cannot tell which one."

Many (most?) of my rips are just one (or a few) tracks from compilation CDs, because they are tracks I don't already have, but the rest of the tracks on the CD are ones I do already have.

Well, you could use the feature only when you really need/want to recover a track that dbpoweramp can't

I don't like the idea of being forced to rip an entire CD just to 'recover' the one or two tracks I want, especially when dbPowerAmp already does such a spiffing job of being able to correctly rip damaged CDs, even if they are not in AccurateRip.

Your not forced to use the feature, but if you have a disc with tracks dBpoweramp can't recover, then you could use the feature if you wanted. Of course you would have to rip the whole disc to use it. You can always go to HA and express that you would like to see a track based system, but its not up to me or spoon.

Also, does this CTDB thing take account of the fact that US and European releases of CDs sometimes have different tracks, or are mastered such that the track offsets on the US/European/Asian pressings are subtly different?
--
BFN
CAD

Yes, it takes into account different pressings. There is more discussion on HA about this.

[Spoon]

Re: CUETools DB Support

The fact it is not track based is a real issue, to make it track based it would have to store 10x more correction data, which would make it un-practical.

[EliC]

Re: CUETools DB Support

Of course if a user only wants to rip a few tracks dBpoweramp could, in the background, rip other tracks IF it cannot recover the selected tracks, then use the correction data, and simply delete the unwanted tracks/data

[sredmyer]

Re: CUETools DB Support

On the surface, this idea certainly seems as though it may have promise. As someone who does CD ripping both as a business and a hobby I think that this would have more value for the hobbyist than the business.

For the business it's all about throughput and quality. IOW I must move as many discs as possible through the process in as short an amount of time as possible without negatively impacting the quality of the results. It sounds like this feature would add the ability to repair the occasional rip that dbPoweramp could not. This is obviously a good thing. Unfortunately, it does this at the expense of throughput which is a bad thing. In my experience, dbPoweramp is able to accurately extract the audio from >90% of the discs I have processed. That means this feature would only add value to <10% of the rips done. Adding this feature would be a no-brainer if it were free...but it appears as though the cost of adding it would be a significantly decreased throughput rate. Note, in the previous sentence I am not referring to cost as in the one time cost of how much we pay Spoon for adding the feature but rather as in the on-going cost of the impact on the time to do the processing.

If this (or something like it) could be made to work in conjunction with AR and on the track level it would truly be valuable. Imagine a system where discs are ripped and verified by AR. Only when a track can not be verified by AR would this correction mechanism kick in. Additionally since the size of the data being corrected would be much smaller (single track vs. whole disc) the amount of data sent from the correction DB to the ripping application would be significantly less and since the correction algorithm would have significantly fewer bytes of data to work with the time required to perform its work would also be much less. So implemented in this way the feature would meet the requirements of the business user by improving quality of the rips whilst at the same time only adding a small amount of time to the process and only when required. However as Spoon has pointed out storing correction data at the track level would require a significantly larger DB. I am not sure about his 10X figure though...seems to me that although there would be many times more pieces of data the size of those pieces would be smaller so the net size difference might not be that great (but I am not familiar with how the correction process works and what data is required so I may be completely wrong on that). At any rate in this day-and-age storage is relatively inexpensive and one could store terabytes of data without incurring a huge cost.

Now from my hobbyist point of view the only thing that matters is getting a truly accurate rip. I don't care if a batch of 100 discs takes all weekend to run. So from this perspective it seems the feature is, again, a no-brainer. My only question is, what would be the % of errors this type of thing would actually be able to correct. IOW if dbPoweramp on its own is able to accurately extract >90% of the audio and this feature were able to fix the other 10% then it is worth the extra time (again to a hobbyist). However if such a feature were only able to correct half of the problems is it worth it? Even as a hobbyist I still have a somewhat practical nature. As mentioned previously I would not care if it took all weekend to run a batch of 100 discs if I was sure that when finished I would have 100% accurate results. However it does not appear (from my reading of this features limitations) as though that would necessarily be the case. Rather it seems as though it would be much more likely that I would improve my accuracy only somewhat.

Now having said all of that, I would be a huge advocate of opening up the DSP architecture to outside developers. I realize that we have some limited ability to do these types of things using the "Run External" DSP. However this feature has limited value as it only allows after the fact manipulation of the extracted data. This feature would be much more useful if it provided a true bi-directional real-time interface with the ripping software. Having this ability would allow me to tailor the way the ripping job is done to match my preferred workflow.

[EliC]

Re: CUETools DB Support

Steve,
I don't think there is any implication that AR would NOT be used. AR would absolutely still be used. If after reading an entire disc there are errors dBpoweramp should even be able to estimate if the number of errors exceeds CTDB ability to do a repair. If, and only if, it should be able to repair would it proceed.

For commercial rippers, there would be 2 areas where throughput could/would be affected:

1. Creating new repair files for the database for discs not already in the database. This can be time and CPU intensive. Of course commercial rippers could skip this step, but this would slow the growth and usefulness of the database.

2. The repair itself can be slow and CPU intensive. This would be much faster if it was track based


The question is, as a commercial user, would you pay on a per repair basis to access the database if it was track based?

Would non-commercial users pay a small annual fee to access a track based database?

Track based would have multiple benefits:
1. ~10x more errors could be repaired per disc
2. Repairs would be much faster
3. Single tracks could be ripped and repaired

[EliC]

Re: CUETools DB Support

from: http://www.hydrogenaudio.org/forums/...howtopic=79882

Of course I would prefer to have the larger database for free, but if the choice is between paying for the bigger database or not having it available, I would pay.

I would really like to see spoon use this database in dbpoweramp. He seemed to imply that in its current configuration he is less likely to. I think this is very unfortunate, but his choice.

I'm just not ready to handle financial aspects. I'm not good at this. So it's better to just keep database costs to a minimum, and hope that storage will be cheap enough in the future.

If Mr. Spoon will decide to support CTDB on condition that it's track based, we'll discuss it.

spoon, I don't know what your thoughts are on adding CTDB support, but it seems as though there may be room to add track based recovery files. I hope the conversation can be moved forward and this great feature can make it into R14.

[EliC]

Re: CUETools DB Support

spoon, I know you hate to comment on wishlist features, but I wondering/hoping if you are talking with Gregory S. Chudov or what changes to the CTDB would make it more appealing for you to add as a feature? Thanks.

[Spoon]

Re: CUETools DB Support

It is not high on our plans to support this.

[EliC]

Re: CUETools DB Support

It is not high on our plans to support this.

Spoon, that's very unfortunate, but not surprising. Again, I think this is one of those rare features that can really set the ripper apart and build even more support among the enthusiast crowd.


Are there any changes to the CTDB that would make it a higher priority for you?


Assuming this is still not a high priority, I would again encourage you to consider opening up the DSP architecture. That way end users/developers can extend the features of the ripper and include features like CTDB.

[EliC]

Re: CUETools DB Support

Fill in the last green box:

http://wiki.hydrogenaudio.org/index.php?title=Comparison_of_CD_rippers

CTDB support is all thats missing (at least by this chart - I have a few more suggestions)

[EliC]

Re: CUETools DB Support

CTDB is now available in EAC. It would be great to see dBpoweramp add this feature as well.

[xTobix]

Re: CUETools DB Support

After reading up on CTDB, mainly due to its resent implementation in EAC 1.0b3 (optional yet!), I believe this can be considered as the logical evolution of (secure) audio ripping.
(See here: http://www.exactaudiocopy.de/en/inde...new/whats-new/)

For me it would offer much, since I have seen the same fail-ratio as reported by "sredmyer" of about 10-15% of my rips. However, implementing this as a second measure approach only once the standard readout failed, might seem practical, but does undermine the whole purpose of the idea after all.

If this gets implemented, which I strongly vote for, it only can be a permanent implementation if once chosen by the individual user (installation/version/plug-in ?).

If, in general, fail rates could be diminished to 5% or better this would be a considerable benefit.
I suppose time will tell and so far, it is still under major development.

Hope the best!