Ripping HDCDs: Best practices?

[tewill]

I apologize in advance for this lengthy post and its wordy questions, but after a lot of reading I'm still unsure what the best general approach is for ripping HDCDs, and from what I can tell there is no FAQ or definitive best practices recommended by illustrate. This might be a bit pedantic, but I'm really hoping those of you who know a lot about this can help me determine best practices in their absence.

What I've found so far suggests when ripping HDCDs you should either:


A. Rip them as-is - to a 16 bit FLAC - and let your device handle HDCD decoding.

B. Rip them using the HDCD DSP (to a 24 bit FLAC), with the +6 dB Amplification *unchecked.*

C. Rip them using the HDCD DSP, with the +6 dB Amplification checked.

D. Rip them using the HDCD DSP, don't check the +6 dB, but add the Volume Normalize DSP with the ReplayGain (track gain) option set.


I'm uncertain which of these is the better choice, and had some questions:

1. I understand C & D above are suggested b/c the 24 bit FLAC would otherwise be significantly quieter, and that D is recommended over C because the amplification will be customized to the specific track (and minimize clipping), but I thought it's considered a big no-no to apply gain by modifying the audio in the file itself? [Edit: My understanding is that C & D are also suggested due to situations where ReplayGain tags can't be used to otherwise increase the volume, when the HDCD DSP is used. And, BTW, the +6 dB amplification option is checked by default when using the HDCD DSP.]

2. It seems like B, C, & D are primarily relevant to those who play FLACs - as opposed to using them as archives - but if you wanted to apply the HDCD DSP, wouldn't a good best practice be to first rip to FLAC without the HDCD DSP, then create a second copy for listening via B, C, or D (using dMC), so that you still have an unadulterated copy?

3. Is option D really better than C? I've still seen clipping on such rips (although much less), and it seems in theory that amplifying all the tracks with track gain would remove any nuances within the album, e.g. where some tracks are intentionally quieter. Alternatively, it seems the +6 dB option would retain the nuances, but then greatly increase the chances of clipping (most I've ripped had *lots* of clipping). Could this provide reason to instead go with option A or B?

4. Is the HDCD DSP even necessary? My understanding is that the sole reason to apply it is in case you have a device that cannot decode HDCDs. But, do most modern devices decode HDCDs? In other words, is the proportion of devices that can't decode HDCDs high enough to justify using the HDCD DSP as a best practice, or is the opposite the case? Is there another reason to use the HDCD DSP, e.g. does it somehow ensure the sound is superior over device-dependent decoding?

5. Is it true that some tracks in an album might be HDCD encoded whereas others are not, and that only certain HDCD features might be applied (and not others) to any given track? If these are the case, could they further support that the best practice should be A above, as it otherwise might be challenging to discover which tracks need amplification, etc. and you should instead just let your device determine what needs to be done (just as it would if you were playing the HDCD directly)?

6. And, what about transcoding to a lossy format from a FLAC that was ripped from an HDCD? I've read that mp3s have no fixed bit depth, with the encoding stored as floating point, that most decoders decode to 16 bit, and if they do have the ability to decode to different bit rates they need to be configured to do so (and I imagine manually set back to 16 bit afterwards). That said, does the audio quality in an mp3 transcoded from a FLAC ripped from an HDCD depend upon the method (A, B, C, or D) used to rip the FLAC? For example, is the quality improved when using A, as LAME would see the FLAC as 16 bit, compared to B, C, or D, where LAME sees the FLAC as 24 bit?

7. As an aside, I think it would be great if CD Ripper flagged FLACs ripped from HDCDs, because if you don't know they're from HDCDs, the only way to find out is to use dMC to "convert" them to FLAC but with the HDCD DSP applied, to see if the outputted files are 24 bit. I saw that someone recommended adding "HDCD" to the Source tag instead of "CD."

Thanks!

[garym]

Re: Ripping HDCDs: Best practices?

Can't answer, but two things. One, replaygain DSP simply adds tags. It doesn't change audio at all. Resulting flac file is still lossless aydio rip. Then players use these tags to adjust volume. Second, when adding RG tags, you can add track gain and album gain. My players (squeeezeboxes or foobar2000) can use either. To maintain inter-track, intra-album volume differences, use album RG.

[tewill]

Re: Ripping HDCDs: Best practices?

In this context I'm referring to ReplayGain being used as an option for the Volume Normalize DSP. From the Volume Normalize help: "ReplayGain: a more advanced normalization, the average loudness is calculated and the volume is adjusted. This method does not use ID Tags to store ReplayGain values, rather the audio is adjusted."

[garym]

Re: Ripping HDCDs: Best practices?

In this context I'm referring to ReplayGain being used as an option for the Volume Normalize DSP. From the Volume Normalize help: "ReplayGain: a more advanced normalization, the average loudness is calculated and the volume is adjusted. This method does not use ID Tags to store ReplayGain values, rather the audio is adjusted."

Ok. Bad idea to do this in my opinion. Rips are no longer lossless.

[Mike48]

Re: Ripping HDCDs: Best practices?

tewill,

I make no claims about best practices, but I use your method "C". Not every playback device can decode HDCD, so why not take care of that while ripping? I increase the volume because it is a form of leveling that can be done with no loss of quality. And yes, my library is primarily for listening. I use the physical CD as the archive (though I make frequent backups).

I do not believe that any discs have been made that are only part HDCD. The exception might be a demo disc comparing HDCD to non-HDCD. I would expect that dBpa works on a track-by-track basis, though I don't know that for a fact.

I would want the full dynamic range (24 bits) sent to the lossy codec. However, this is somewhat academic, because the point of using a lossy codec is convenience, not sound quality. When making mp3 copies, such things don't really bother me. I use the dBpa DSP facilities to reduce bit depth while converting.

I agree, it would be nice for dBpa to flag HDCD files as such in the tags.

[pablogm123]

Re: Ripping HDCDs: Best practices?

I do prefer the option A, to have a backup copy in lossless format. Then, I would analyze the music library (at least the freshly ripped portion) by the Foobar 2000 foo_hdcd plug-.in, and I would encode a second copy applying option C by the music converter and DSP HDCD with +6db amplification checked. I would tag the decoded album this way: [NAME OF ORIGINAL ALBUM] HDCD.

[Porcus]

Re: Ripping HDCDs: Best practices?

@ tewill:


Yeah, a FAQ ...

If you are ripping to lossless, then my recommendation is option A: keep the CD intact and decode upon playback. Reasons posted at http://forum.dbpoweramp.com/showthre...l=1*post119732 . One of the reasons is that you do not need the physical CD to decode HDCD. You can always decode later, but decoding is irreversible.


A few more notes re your options:
(A): If you care about the 'transient filter', then there is no software implementation. Only way to have that processed - if such one exists! - is to feed the unprocessed signal to a HDCD-aware DAC.
(B) / (C): Here starts trouble: the tracks on a CD may not all have the same flag. For this reason, I would in the least want the entire CD ripped and scanned before considering HDCD decoding.


Re ReplayGain: Beware the difference between the ReplayGain and the ReplayGain Apply DSPs; the latter alters the audio, the former writes tags.


Re your consideration *2: I'd say that options B, C & D are relevant to those who do not play lossless. Lossless files can be decoded afterwards, lossy files (likely!) not.

Re your 3: ReplayGain should normally be applied on album basis. Then you might be in trouble of some are boosted and others not. A player like foobar2000 allows for manual ReplayGain adjustment.

Re your 4: If your software/hardware cannot playback HDCD, then you can always apply the HDCD DSP afterwards.

Re your 5: Yes, true, and I agree with the consequence you draw; again, you can do HDCD decoding later if you want to.

Re your 6: If you transcode to lossy - supposing you want the added dynamics from e.g. the peak extension - then I would decode first.

Re your 7: The hdcd.exe can be used to scan and report without any actual conversion. Myself I rather use foobar2000 with the HDCD component.

[tewill]

Re: Ripping HDCDs: Best practices?

tewill,

I make no claims about best practices, but I use your method "C". Not every playback device can decode HDCD, so why not take care of that while ripping? I increase the volume because it is a form of leveling that can be done with no loss of quality. And yes, my library is primarily for listening. I use the physical CD as the archive (though I make frequent backups).

My main concern with option C is that, in most cases when I've added the default +6 dB, the tracks showed quite a lot of clipping. Admittedly I had difficulty hearing a difference between tracks with and without this clipping, and not all tracks I've done this to have shown clipping, but again my ideal goal is to develop best practices so I don't have to use a different approach for each HDCD I rip. Currently I use my FLACs as an archive and put my CDs in storage, so am reluctant to make any changes that directly affect the lossless audio. Personally I'm all about the "rip once" philosophy, and shudder at the thought of having to dig out CDs to re-rip. So, even if I were to use the HDCD DSP, I'd likely create a second FLAC copy with it applied.

I do not believe that any discs have been made that are only part HDCD. The exception might be a demo disc comparing HDCD to non-HDCD. I would expect that dBpa works on a track-by-track basis, though I don't know that for a fact.

I've read some posts by Porcus (one of which he references above) and although I don't have this knowledge about HDCDs, he argues that - if I'm understanding him correctly - there are HDCD features that can be lost by using the HDCD DSP as well as that there are albums (or even individual tracks, I believe) that can be flagged as HDCD yet don't contain the increased bit depth.

[tewill]

Re: Ripping HDCDs: Best practices?

I do prefer the option A, to have a backup copy in lossless format. Then, I would analyze the music library (at least the freshly ripped portion) by the Foobar 2000 foo_hdcd plug-.in, and I would encode a second copy applying option C by the music converter and DSP HDCD with +6db amplification checked. I would tag the decoded album this way: [NAME OF ORIGINAL ALBUM] HDCD.

I'm not familiar with this plug-in; it looks like it functions to decode HDCDs for the foobar2000 player, but I'm assuming it can prominently flag a 16 bit stream as HDCD, and therefore offer an easier way to discover which 16 bit FLACs are HDCD encoded?

I haven't yet experimented much with option B, using the HDCD DSP without any amplification, and am curious if the volume is so low that it becomes prohibitive. That is, I usually play my audio files without using ReplayGain (although I do write the tags), as I prefer the sound without it and don't mind turning the volume up or down a bit if need be. But, if option B makes the volume so low that turning up the volume can't get it to a listenable level, then it would seem that some amplification would be necessary (or else you'd need to have ReplayGain tags written and it turned on in your player). Although Spoon in this post recommends option D over C, I imagine b/c of the potential for clipping, I'm however equally concerned about losing the dynamics of the album as a whole by using option D. And again, my hope is to not have to change my ripping process for each individual HDCD, e.g. whether a particular HDCD is loud enough to use no amplification, quiet enough to use the +6 dB amplification, or balanced enough to use ReplayGain track gain info to amplify each track.

I do definitely agree with you about having a lossless backup.

[pablogm123]

Re: Ripping HDCDs: Best practices?

foo_hdcd can analyze already ripped files, and reports if contains standard/HDCD content. If needed, you can encode a decoded set of files (for example, for a hardware player which doesn't support HDCD) by the HDCD DSP.

[Porcus]

Re: Ripping HDCDs: Best practices?

tewill: the foobar2000 plugin is here: http://www.foobar2000.org/components/view/foo_hdcd

Personally I'm all about the "rip once" philosophy, and shudder at the thought of having to dig out CDs to re-rip.

Yeah ... I still haven't re-ripped the CDs I ripped with the HDCD converting to 24 bits :-( Only a few, which were readily available.

I've read some posts by Porcus (one of which he references above) and although I don't have this knowledge about HDCDs, he argues that - if I'm understanding him correctly - there are HDCD features that can be lost by using the HDCD DSP as well as that there are albums (or even individual tracks, I believe) that can be flagged as HDCD yet don't contain the increased bit depth.

- The transient filters have not been implemented by software. If you decode once, you will lose the ability to later decode those filters. I don't know what those filters do, and whether they are good for anything. A few words here: http://www.hydrogenaudio.org/forums/...dpost&p=737725. (Oh, by the way: I don't even know what happens if the HDCD is applied, but there is no peak extension and no 6dB reduction, only low-level extension; could it then be that the HDCD flags are not destroyed, and the LLE is doubled if it is passed on to a HDCD-aware DAC? I have no idea.)

- There are albums which are flagged as HDCD, but don't use HDCD features. (Again, see previous link.) From my own collection: Bruce Springsteen's «Tracks» box: of the four CDs, the foobar2000 component flags three tracks with one feature «intermittent» (which I don't really know what means in this case, as the correction is 0dB), and that's it. Using foobar2000 with foo_hdcd. dBpoweramp with HDCD.exe created 24-bits file of these (I re-ripped).

- Yes, there are albums where some but not all tracks have HDCD codes, and not only various artists compilations: I ripped Ani DiFranco / Utah Phillips: Fellow Workers with dBpoweramp using the HDCD.exe DSP. Of the eighteen tracks, numbers 4, 5, 6, 10, 13, 14 and 15 ended up as 24 bits files, the nine others did not.

[tewill]

Re: Ripping HDCDs: Best practices?

Yeah, a FAQ ...

Agreed, and I've seen you appeal for one in other posts. I think it's especially important as I imagine many new to dBpoweramp or ripping audio might believe that applying the HDCD DSP is a requirement to play HDCD audio on any device. And, I could see those same people easily leaving the +6 dB amplification option checked as well (without knowing whether they truely want to), as it's currently the default. To add to the confusion, as I mentioned above Spoon recommends that the default not be used.

A few more notes re your options:
(A): If you care about the 'transient filter', then there is no software implementation. Only way to have that processed - if such one exists! - is to feed the unprocessed signal to a HDCD-aware DAC.
(B) / (C): Here starts trouble: the tracks on a CD may not all have the same flag. For this reason, I would in the least want the entire CD ripped and scanned before considering HDCD decoding.

Both of these points therefore pointing towards option A, ripping without the HDCD DSP, as the best option.

Re your 4: If your software/hardware cannot playback HDCD, then you can always apply the HDCD DSP afterwards.

Good point to make, as I think many who apply the HDCD DSP do so for the theoretical case in which they'd want to play the audio on a device that can't decode HDCDs. This reasoning could be signficantly off, however, if all of one's current devices can decode HDCD and there is the risk of certain HDCD features being lost by applying the HDCD DSP.

Re your 6: If you transcode to lossy - supposing you want the added dynamics from e.g. the peak extension - then I would decode first.

This is something about which I haven't yet found any information, but what you recommend at least makes sense. I'll have to play around a bit, as I'm wondering if the HDCD DSP can be applied when transcoding a FLAC to an mp3 without affecting the source FLAC, or if you'd have to make a copy of the FLAC with the HDCD DSP applied before you transcode. That said, it does still bring up the issue of whether to amplify the decoded HDCD audio and if so whether to use option C or D (and, at least for option D, whether you should do it before or after transcoding). I have read some posts where people have been able to hear the difference the clipping makes with the default +6 dB amplification option checked (BTW, I'm not sure why the +6 dB amount is used in particular - as esp. I'm not sure what is meant by the "align 20 bit result in upper 24 bit range" that is written next to the option - or why it was decided that it would be checked by default.

[tewill]

Re: Ripping HDCDs: Best practices?

So I've discovered more about HDCDs and ripping HDCDs, primarily from this thread:

http://www.hydrogenaudio.org/forums/index.php?showtopic=79427

Here's a summary of what I so far understand to be the case, but again I'm trying to figure this out myself and am relying on the inferred expertise of others:

Regarding HDCDs in general:

1. There are three primary HDCD features that can be enabled:

a. Peak Extend (PE)
b. Low-Level Extension (LLE)
c. Transient Filter (TF)

2. You can use foobar2000 with the HDCD decoder component installed to see if an album is flagged as HDCD as well as which of these three features are enabled for any given track.

3. HDCD decoding is only needed if the PE or LLE features are enabled.

4. That said:

a. PE is either on or off for an entire album. But, it doesn't do anything until the peak level of a track reaches -9 dBFS, which some older [re-mastered] albums might not reach (because they're quieter). So, PE mainly functions on albums that were mastered on the louder side.

b. LLE can turn on or off during any given track, but doesn't activate until the signal level drops below -45 dBFS, and most tracks don't reach this low level except during silence. So, in many cases HDCD decoding isn't even necessary for tracks with LLE enabled.

5. Albums can have PE and LLE disabled - even have all HDCD features disabled - and still be flagged as HDCD.

6. Such albums can still be flagged as HDCD because during production, when analog audio is converted to digital, certain converters always insert an HDCD "subcode", regardless of whether it is needed.

7. Therefore: HDCD encoding is primarily of value when PE is enabled and the album isn't mastered on the quiet side, but can in general be seen as a marketing ploy.


Regarding decoding HDCDs:

1. Decoding HDCDs will effectively halve the volume, by -6 dB.

2. However, this volume reduction is only necessary if PE is enabled.

3. Nonetheless, this volume reduction is applied when decoding all HDCDs, regardless of which HDCD features are enabled.

4. So, the +6 dB amplification is only really needed for tracks ripped from HDCDs that don't have PE enabled and when you're not using ReplayGain tags to increase the volume of such tracks.

a. A decoded HDCD with PE enabled and without added amplification will likely still be a little quieter, but applying +6 dB amplification to such a decoded HDCD risks taking the volume too high and might result in clipping.

b. BTW, you don't want to use ReplayGain on your player if you are playing a file ripped from an HDCD that has PE enabled, as volume reduction by ReplayGain could negate the PE feature (I think some software players even disable HDCD decoding if ReplayGain is turned on). Technically this advice would also apply to rips from HDCDs with the LLE feature enabled, but again for many tracks the LLE feature is amplifying silence and is therefore questionable in value.

5. However, in the case when PE is enabled but is effectively not doing anything because of a low track peak level, it seems the +6 dB increase could be necessary, but I'm not sure of this.

6. Therefore: If PE and LLE are disabled on an HDCD - and in practice actually if just PE is disabled - and you decode the HDCD, all it essentially does is lower the volume. And, from personal experience, you might be surprised by your HDCDs that have PE or all three HDCD features disabled.


Regarding transcoding HDCDs to a lossy format:

1. This refers to the foobar2000 player, but the conclusion seems to also be relevant to dMC:

http://www.hydrogenaudio.org/forums/index.php?showtopic=79427&st=175&p=737575&*entry737575

"The HDCD decoder filter always outputs 20 bits precision. It is up to the user to set their output device to 24 bits or greater for playback. Or, if using the HDCD decoder when transcoding to another format, it's up to the user to transcode to a 24 bit format or greater, or some lossy format that accepts 24 bit or floating point sample data.

This is meaningless for portable players, since most of them only have 16 bit DACs anyway."

2. It appears that it will work to apply the HDCD DSP when transcoding from a FLAC to an mp3 (i.e. if you don't want to apply the HDCD DSP to the FLAC but do want it applied prior to transcoding, and instead of first creating a second copy of the FLAC with the HDCD DSP applied then transcoding to mp3 from that second copy), which could mean less work if this is something you'd want to do, e.g. if PE is enabled.

[garym]

Re: Ripping HDCDs: Best practices?

So I've discovered more about HDCDs and ripping HDCDs, primarily from this thread:

http://www.hydrogenaudio.org/forums/index.php?showtopic=79427

Here's a summary of what I so far understand to be the case, but again I'm trying to figure this out myself and am relying on the inferred expertise of others:
[...detail removed....]

Thanks for taking the time to summarize all this. Very useful.

[Porcus]

Re: Ripping HDCDs: Best practices?

- You cannot trust PE being all-or-nothing on an album. It should be that way, but it isn't necessarily so, especially not with compilations.

- Remasters are often louder. Older albums mastered with headroom, that's often the "non-remasters".

- The -9 is post-decoding. If you consider the raw undecoded output, it is -3. That means there are more of those, but still few.


The PE works by – at encoding – squeezing the top 9 dB (dBFS, not highest in the signal) into the top 3 dB of the encoded signal that ends up on the CD. Without decoding, that's what you get; with decoding, the top 3 dB will be expanded into the bracket [-3 to +6] decibels; that would introduce clipping, so that one reduces the entire signal by 6 dB to get peak down to 0. Then the signal is on par with everything else which was peak-normalized in the first place.

If there is no PE, the signal could be left as-is.

"The HDCD decoder filter always outputs 20 bits precision. It is up to the user to set their output device to 24 bits or greater for playback. Or, if using the HDCD decoder when transcoding to another format, it's up to the user to transcode to a 24 bit format or greater, or some lossy format that accepts 24 bit or floating point sample data.

This is meaningless for portable players, since most of them only have 16 bit DACs anyway."

The latter is a bit ambiguous - what's this "this" which is meaningless? It is not meaningless to decode a signal which uses PE, as it reverses compression. It is of course meaningless to try to stuff the last eight bits down the throat of a DAC that can only chew 16.