Ripping HDCDs: Best practices?

[garym]

Re: Ripping HDCDs: Best practices?

Quick (I think, or maybe just naive) question. If I have an HDCD that was ripped to FLAC without use of the HDCD dsp, is it true that I can:

1. convert from FLAC to FLAC, adding the HDCD DSP in this conversion, and not *also* applying ReplayGain
2. Afterwards apply ReplayGain to the newly created FLAC files

That is, it is NOT important that I apply HDCD DSP during *ripping* as opposed to simply applying it during a FLAC to FLAC conversion.

[garym]

Re: Ripping HDCDs: Best practices?

Quick (I think, or maybe just naive) question. If I have an HDCD that was ripped to FLAC without use of the HDCD dsp, is it true that I can:

1. convert from FLAC to FLAC, adding the HDCD DSP in this conversion, and not *also* applying ReplayGain
2. Afterwards apply ReplayGain to the newly created FLAC files

I realize that these new FLAC files would no longer be lossless, but I'd be able to play them with non-HDCD players. And to be clear, if I play my original FLAC files (that were NOT ripped using HDCD DSP) through a player that is capable of handling HDCD decoding, they should play as HDCD disks.

[EliC]

Re: Ripping HDCDs: Best practices?

Spoon, any chance we can have these options build into the HDCD DSP?

- Only apply HDCD DSP if:
( ) Peak Extend (PE) is enabled
( ) Low-Level Extension (LLE) is enabled
( ) Transient Filter (TF) is enabled

- Apply HDCD DSP, recalculate replaygain/EBU R128, then apply

[tewill]

Re: Ripping HDCDs: Best practices?

Now I need to figure out which discs are HDCDs and go back and re-rip them...

Just in case you don't already know this, you can use dBpoweramp for this purpose, by using it to see what the sample size is for lossless files to which you applied the HDCD DSP. If they're flagged as 24 bit, they were ripped from HDCDs. It's still a bit of a pain, but I find using the Batch Converter can be a bit easier: once added to the Batch Converter, you can select "List" from the menu then add "Sample Size" as a column if it isn't already there.

When transcoding, I like to apply replaygain to my lossy files. Is there a one step way to detect HDCD, apply the HDCD DSP, and then apply replaygain?

A few things here: First, from what I've found, when transcoding to lossy you should only apply the HDCD DSP if the PE feature is enabled, and, if it is, you shouldn't use ReplayGain because it could affect the benefits of peak extension. Yes, if you're transcoding from HDCD-encoded lossless audio the resulting lossy audio will be a bit quieter, but it won't be by much (especially relative to those transcoded from HDCD-encoded files that don't have PE enabled) and the benefit is that it should sound better than it would if the HDCD wasn't decoded. That said, if it is an issue that they're quieter, it might not be worthwhile to decode the HDCD to begin with. Second, I have found that you can apply the HDCD DSP while transcoding and it will work to decode HDCD audio for the lossy files and not affect the audio in the lossless source files. However, if you still wanted to apply the ReplayGain DSP, from what I've found you can't do so during transcoding, as it will base its values upon the un-decoded audio; so you therefore would have to apply the ReplayGain DSP as part of a second step, after the files are transcoded to a lossy format.

[EliC]

Re: Ripping HDCDs: Best practices?

When transcoding, I like to apply replaygain to my lossy files. Is there a one step way to detect HDCD, apply the HDCD DSP, and then apply replaygain?

[EliC]

Re: Ripping HDCDs: Best practices?

I finally stopped using the HDCD DSP. As an added benefit, my CD ripping has increased probably >10x. Something about the implementation of that DSP really slows down ripping.

Now I need to figure out which discs are HDCDs and go back and re-rip them...

[tewill]

Re: Ripping HDCDs: Best practices?

Circling back on this, here are the best practices I came up with for ripping HDCDs:

1. In general, rip HDCDs to a lossless format and don't apply the HDCD DSP.
   
   1. You also shouldn't write ReplayGain tags on these lossless files because:
      
      1. ReplayGain can negatively affect the PE and LLE features.
         
         
      2. When you use the ReplayGain DSP, values are calculated based on the un-decoded HDCD audio, and these values won't be valid once the files are decoded by your device.
   
   
2. That said, if you know for a fact you'll be playing these lossless files on devices that can't decode HDCD-encoded files:
   
   1. Use foobar2000 to analyze the files and see which HDCD features are enabled.
      
      
   2. If Peak Extend is enabled, apply the HDCD DSP to the files, with the +6 db amplification option *unchecked*, and don't write ReplayGain tags.
      
      
   3. If Peak Extend is not enabled, leave the files as they are and do not apply the HDCD DSP.
      
      
   4. NOTE: Unless you don't mind the possibility of re-ripping CDs in the future, if you choose to apply the HDCD DSP to your lossless files consider first making archive copies, as the HDCD DSP will physically alter the audio data.
   
   
3. If you plan on transcoding the lossless files to a lossy format:
   
   1. Same as above, if PE is enabled apply the HDCD DSP while transcoding, with the +6 db amplification option *unchecked*. Otherwise don't apply the HDCD DSP while transcoding.
      
      1. NOTE: Applying the HDCD DSP *while* transcoding doesn't appear to affect the source lossless files. You could also apply the HDCD DSP to the lossless files prior to transcoding, but again note that doing so will physically alter the audio data in those files.

[Mike48]

Re: Ripping HDCDs: Best practices?

tewill -- Thanks for the detailed summary. I appreciate it very much -- it's quite useful -- and I have in fact changed the defaults I use when decoding.

[tewill]

Re: Ripping HDCDs: Best practices?

That should not happen. Either there has been some decoding done (and the processing has sent the signal beyond 0dBFS), or they simply sound that way when not decoded. (Since it then is dynamically compressed, that is maybe what you hear.)

I checked again, and I am mistaken - there isn't clipping in my un-decoded lossless test tracks, but there is clipping when I transcode the un-decoded lossless files to mp3 as well as if I first apply the HDCD DSP with the +6 dB amplification and then transcode to mp3 (PE was disabled for all tracks). So it appears my point is only relevant when transcoding, i.e. when transcoding HDCD-encoded files to mp3, if they're mastered on the louder side they might show clipping and you might want to consider using the HDCD DSP without the +6 dB amplification prior to transcoding.

So, still a small pain if you're transcoding, as it seems you can't take a blanket approach (e.g. apply the HDCD DSP prior to transcoding only if PE is enabled or always apply the HDCD DSP prior to transcoding, but disable the +6 dB amplification), as I've also seen some transcoded tracks for which I first applied the HDCD DSP without the +6 dB amplification and the resulting mp3s were prohibitively quiet (i.e. even with the volume all the way up they were still too quiet).

Did you use RG on those files? Checked your settings? (If you use fb2k: prevent clipping according to peak? Amplified those which are not RG'ed?)

I don't use ReplayGain, and haven't used any player settings to correct clipping; I'm seeing the clipping by looking at the waveform in Audacity.

BTW, all this probably really is pedantic [for me] because for the most part I have difficulty hearing differences between HDCD tracks with different treatments. Besides noticing the obvious volume decrease when applying the HDCD DSP without amplification, the only other difference I've noticed was that when transcoding to mp3 from tracks with PE enabled, the mp3s did seem to sound better if the HDCD DSP was applied without the +6 dB first, as opposed to transcoding without the HDCD DSP. But even then I really had to strain to hear a difference. I also have trouble hearing issues caused by the clipping I mentioned in the transcoded mp3s, so my guess is that after all this detail work I'll end up with a pretty simplistic approach to dealing with HDCDs.

Thanks for the help though!

[Porcus]

Re: Ripping HDCDs: Best practices?

When test-ripping some of my HDCDs, I found that some for which PE was disabled were still mastered on the loud side (many of which weren't remasters) and showed clipping when the HDCD wasn't decoded.

That should not happen. Either there has been some decoding done (and the processing has sent the signal beyond 0dBFS), or they simply sound that way when not decoded. (Since it then is dynamically compressed, that is maybe what you hear.)

or write ReplayGain tags if you use ReplayGain).

Did you use RG on those files? Checked your settings? (If you use fb2k: prevent clipping according to peak? Amplified those which are not RG'ed?)

[tewill]

Re: Ripping HDCDs: Best practices?

- Remasters are often louder. Older albums mastered with headroom, that's often the "non-remasters".

If there is no PE, the signal could be left as-is.

When test-ripping some of my HDCDs, I found that some for which PE was disabled were still mastered on the loud side (many of which weren't remasters) and showed clipping when the HDCD wasn't decoded. I was previously thinking that HDCDs without PE enabled might not need to be decoded, but after seeing this it made me consider the possibility that some HDCDs with PE disabled might be purposely mastered on the louder side with the assumption that their volume would be reduced by 6 dB when played (because they'd be decoded). Point being I'm not sure if it can be said that across the board it is unnecessary to decode HDCDs with PE disabled, but this is mainly only a concern to begin with if you play on devices that can't decode HDCD or if you are transcoding. Regardless, it's still a bit of a pain, as in such cases it could mean you'd need to do the extra work of seeing whether an HDCD with PE disabled was mastered on the louder side (and decode the HDCD, but don't add any amplification) or mastered on the quieter side and not decode the HDCD (or decode it using the HDCD DSP and either add the +6 dB amplification or write ReplayGain tags if you use ReplayGain).

The latter is a bit ambiguous - what's this "this" which is meaningless? It is not meaningless to decode a signal which uses PE, as it reverses compression. It is of course meaningless to try to stuff the last eight bits down the throat of a DAC that can only chew 16.

This is a direct quote from the referenced post. I took it to mean your latter point.

[Porcus]

Re: Ripping HDCDs: Best practices?

- You cannot trust PE being all-or-nothing on an album. It should be that way, but it isn't necessarily so, especially not with compilations.

- Remasters are often louder. Older albums mastered with headroom, that's often the "non-remasters".

- The -9 is post-decoding. If you consider the raw undecoded output, it is -3. That means there are more of those, but still few.


The PE works by – at encoding – squeezing the top 9 dB (dBFS, not highest in the signal) into the top 3 dB of the encoded signal that ends up on the CD. Without decoding, that's what you get; with decoding, the top 3 dB will be expanded into the bracket [-3 to +6] decibels; that would introduce clipping, so that one reduces the entire signal by 6 dB to get peak down to 0. Then the signal is on par with everything else which was peak-normalized in the first place.

If there is no PE, the signal could be left as-is.

"The HDCD decoder filter always outputs 20 bits precision. It is up to the user to set their output device to 24 bits or greater for playback. Or, if using the HDCD decoder when transcoding to another format, it's up to the user to transcode to a 24 bit format or greater, or some lossy format that accepts 24 bit or floating point sample data.

This is meaningless for portable players, since most of them only have 16 bit DACs anyway."

The latter is a bit ambiguous - what's this "this" which is meaningless? It is not meaningless to decode a signal which uses PE, as it reverses compression. It is of course meaningless to try to stuff the last eight bits down the throat of a DAC that can only chew 16.

[garym]

Re: Ripping HDCDs: Best practices?

So I've discovered more about HDCDs and ripping HDCDs, primarily from this thread:

http://www.hydrogenaudio.org/forums/index.php?showtopic=79427

Here's a summary of what I so far understand to be the case, but again I'm trying to figure this out myself and am relying on the inferred expertise of others:
[...detail removed....]

Thanks for taking the time to summarize all this. Very useful.

[tewill]

Re: Ripping HDCDs: Best practices?

So I've discovered more about HDCDs and ripping HDCDs, primarily from this thread:

http://www.hydrogenaudio.org/forums/index.php?showtopic=79427

Here's a summary of what I so far understand to be the case, but again I'm trying to figure this out myself and am relying on the inferred expertise of others:

Regarding HDCDs in general:

1. There are three primary HDCD features that can be enabled:

a. Peak Extend (PE)
b. Low-Level Extension (LLE)
c. Transient Filter (TF)

2. You can use foobar2000 with the HDCD decoder component installed to see if an album is flagged as HDCD as well as which of these three features are enabled for any given track.

3. HDCD decoding is only needed if the PE or LLE features are enabled.

4. That said:

a. PE is either on or off for an entire album. But, it doesn't do anything until the peak level of a track reaches -9 dBFS, which some older [re-mastered] albums might not reach (because they're quieter). So, PE mainly functions on albums that were mastered on the louder side.

b. LLE can turn on or off during any given track, but doesn't activate until the signal level drops below -45 dBFS, and most tracks don't reach this low level except during silence. So, in many cases HDCD decoding isn't even necessary for tracks with LLE enabled.

5. Albums can have PE and LLE disabled - even have all HDCD features disabled - and still be flagged as HDCD.

6. Such albums can still be flagged as HDCD because during production, when analog audio is converted to digital, certain converters always insert an HDCD "subcode", regardless of whether it is needed.

7. Therefore: HDCD encoding is primarily of value when PE is enabled and the album isn't mastered on the quiet side, but can in general be seen as a marketing ploy.


Regarding decoding HDCDs:

1. Decoding HDCDs will effectively halve the volume, by -6 dB.

2. However, this volume reduction is only necessary if PE is enabled.

3. Nonetheless, this volume reduction is applied when decoding all HDCDs, regardless of which HDCD features are enabled.

4. So, the +6 dB amplification is only really needed for tracks ripped from HDCDs that don't have PE enabled and when you're not using ReplayGain tags to increase the volume of such tracks.

a. A decoded HDCD with PE enabled and without added amplification will likely still be a little quieter, but applying +6 dB amplification to such a decoded HDCD risks taking the volume too high and might result in clipping.

b. BTW, you don't want to use ReplayGain on your player if you are playing a file ripped from an HDCD that has PE enabled, as volume reduction by ReplayGain could negate the PE feature (I think some software players even disable HDCD decoding if ReplayGain is turned on). Technically this advice would also apply to rips from HDCDs with the LLE feature enabled, but again for many tracks the LLE feature is amplifying silence and is therefore questionable in value.

5. However, in the case when PE is enabled but is effectively not doing anything because of a low track peak level, it seems the +6 dB increase could be necessary, but I'm not sure of this.

6. Therefore: If PE and LLE are disabled on an HDCD - and in practice actually if just PE is disabled - and you decode the HDCD, all it essentially does is lower the volume. And, from personal experience, you might be surprised by your HDCDs that have PE or all three HDCD features disabled.


Regarding transcoding HDCDs to a lossy format:

1. This refers to the foobar2000 player, but the conclusion seems to also be relevant to dMC:

http://www.hydrogenaudio.org/forums/index.php?showtopic=79427&st=175&p=737575&*entry737575

"The HDCD decoder filter always outputs 20 bits precision. It is up to the user to set their output device to 24 bits or greater for playback. Or, if using the HDCD decoder when transcoding to another format, it's up to the user to transcode to a 24 bit format or greater, or some lossy format that accepts 24 bit or floating point sample data.

This is meaningless for portable players, since most of them only have 16 bit DACs anyway."

2. It appears that it will work to apply the HDCD DSP when transcoding from a FLAC to an mp3 (i.e. if you don't want to apply the HDCD DSP to the FLAC but do want it applied prior to transcoding, and instead of first creating a second copy of the FLAC with the HDCD DSP applied then transcoding to mp3 from that second copy), which could mean less work if this is something you'd want to do, e.g. if PE is enabled.

[tewill]

Re: Ripping HDCDs: Best practices?

Yeah, a FAQ ...

Agreed, and I've seen you appeal for one in other posts. I think it's especially important as I imagine many new to dBpoweramp or ripping audio might believe that applying the HDCD DSP is a requirement to play HDCD audio on any device. And, I could see those same people easily leaving the +6 dB amplification option checked as well (without knowing whether they truely want to), as it's currently the default. To add to the confusion, as I mentioned above Spoon recommends that the default not be used.

A few more notes re your options:
(A): If you care about the 'transient filter', then there is no software implementation. Only way to have that processed - if such one exists! - is to feed the unprocessed signal to a HDCD-aware DAC.
(B) / (C): Here starts trouble: the tracks on a CD may not all have the same flag. For this reason, I would in the least want the entire CD ripped and scanned before considering HDCD decoding.

Both of these points therefore pointing towards option A, ripping without the HDCD DSP, as the best option.

Re your 4: If your software/hardware cannot playback HDCD, then you can always apply the HDCD DSP afterwards.

Good point to make, as I think many who apply the HDCD DSP do so for the theoretical case in which they'd want to play the audio on a device that can't decode HDCDs. This reasoning could be signficantly off, however, if all of one's current devices can decode HDCD and there is the risk of certain HDCD features being lost by applying the HDCD DSP.

Re your 6: If you transcode to lossy - supposing you want the added dynamics from e.g. the peak extension - then I would decode first.

This is something about which I haven't yet found any information, but what you recommend at least makes sense. I'll have to play around a bit, as I'm wondering if the HDCD DSP can be applied when transcoding a FLAC to an mp3 without affecting the source FLAC, or if you'd have to make a copy of the FLAC with the HDCD DSP applied before you transcode. That said, it does still bring up the issue of whether to amplify the decoded HDCD audio and if so whether to use option C or D (and, at least for option D, whether you should do it before or after transcoding). I have read some posts where people have been able to hear the difference the clipping makes with the default +6 dB amplification option checked (BTW, I'm not sure why the +6 dB amount is used in particular - as esp. I'm not sure what is meant by the "align 20 bit result in upper 24 bit range" that is written next to the option - or why it was decided that it would be checked by default.