Mass Removing Leading and Trailing Whitespaces From a Tag Field

[schmidj]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

Excuse me for getting away from the OP's issue, but a couple of comments on using the EBU R128 standard for volume normalization. While it works fairly well with a lot of material, be aware you are using it to do things for which it was never designed. It was designed to simply solve the problem of "loud commercials" in TV broadcasts. They were usually "loud" because they were heavily compressed, little dynamic range (on purpose, by the ad agencies). TV audio processing, particularly in the analog TV era depended on peak limiters to prevent overmodulation. Hence the average "volume" (which to some degree is what the ear hears) of peak normalized heavily compressed audio will always sound "louder" to our ears than the average volume of wide dynamic range peak normalized audio. This is particularly true if part of the wide dynamic range is rare or occasional momentary loud peaks.

The various TV loudness standards are all based on RMS or average volume, not peak volume, so right there they will, within their limitations, provide more constant volume for different source material.

But remember: These standards were developed and tested for TV audio, mostly dialog, not music. They were developed to monitor, and in some cases process audio, not to calculate a "tag" to post-process the audio in a consumer "player", and that creates serious issues with "peaky" high dynamic range material. And they were a pretty crude simulation of human perception of loudness. There is almost no control of psychoacoustic effects, just some mild EQ in the sidechain. (So the screaming barker, even at lower level, will often sound louder than the soft, sweet host.) Finally, this was a "political" solution, to mollify regulatory agencies being flooded with consumer complaints, not a carefully developed technical solution. "Quick and dirty."

Every true compliant TV production loudness processor, either hardware or software, includes a peak limiter, so if the loudness must be increased it "limits" any peaks which would exceed the specified peak level it is set for. But since most, if not all consumer players don't include such limiters, and since the volume normalization gain adjustment is applied in the player, increasing the volume may cause clipping distortion in the consumer's player. To prevent that, the dBpa normalize tag calculation algorythm examines the file for the highest peak and will not allow the volume normalize tag to contain a value that would increase that peak above 0dBFS (the clip point) in the consumer player.

Therefore "peaky" material (most commonly classical and some jazz, but too often these days poorly mastered pop music, will often play back much "softer" than the less peaky recordings.

I've decided that, for my portable (car and headphone) listening, to give up on using the volume normalize tag, and take all the ripped un-normalized tracks and batch process them through one of my professional audio volume normalize plug-ins (which include high quality, fairly inaudible limiters) and create a second master directory of tracks where the audio has been processed and then saved to be at a constant loudness that works with my players. The major issue I have now is that my most used professional audio software strips out all the tags when opening and then saving the files, making the results useless for my players.

[garym]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

I'm jealous of Windows users because Foobar2000 is exactly what I'm looking for. The app for iOS looks promising though and so I'm going to give it a try. Do you know if it supports rating and 'loving' (iTunes/iPhone feature)?



Soundcheck with dBpoweramp's ReplayGain encoding works most of the time but I've noticed that occasionally there are some tracks that are noticeably quieter or louder than the average track. The tags are being written to ITUNNORM XXXX and COMMENT ITUNNORM but as I explained to Spoon the encoding process didn't display a Results screen and all of the ITUNNORM/COMMENT ITUNNORM tags have leading whitespaces. I found a way to remove the whitespace with Jaikoz (haven't figured out how to perform this function with dBpoweramp or PerfectTunes yet) and as a result the volume normalization seems to be effective. For some reason the ITUNNORM XXXX and COMMENT ITUNNORM tags are replicated in the Comments section in iTunes, do you think I can safely delete them without affecting the ReplayGain?

I can't recall whether the app supports loving. It does support rating. But I don't use either function. Also, keep in mind that replaygain/soundcheck, etc. is not perfect. It works very well for volume normalization most of the time, but there will be tracks/albums that still stand out as too loud/quiet. This is more likely to happen if you have albums with lots of dynamic range (classical) or albums so "hot" (loud) that even volume normalization doesn't quite lower the volume enough. I'm no expert on the algorithm used by ReplayGain or Soundcheck.

[garym]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

I'm jealous of Windows users because Foobar2000 is exactly what I'm looking for. The app for iOS looks promising though and so I'm going to give it a try. Do you know if it supports rating and 'loving' (iTunes/iPhone feature)?



Soundcheck with dBpoweramp's ReplayGain encoding works most of the time but I've noticed that occasionally there are some tracks that are noticeably quieter or louder than the average track. The tags are being written to ITUNNORM XXXX and COMMENT ITUNNORM but as I explained to Spoon the encoding process didn't display a Results screen and all of the ITUNNORM/COMMENT ITUNNORM tags have leading whitespaces. I found a way to remove the whitespace with Jaikoz (haven't figured out how to perform this function with dBpoweramp or PerfectTunes yet) and as a result the volume normalization seems to be effective. For some reason the ITUNNORM XXXX and COMMENT ITUNNORM tags are replicated in the Comments section in iTunes, do you think I can safely delete them without affecting the ReplayGain?

leading white space sounded odd, but then I checked some of my mp3 files. The iTunNorm field also has a leading white space. And these iTunNorm values weren't created by dbpa. They were created by mp3tag (based on the ReplayGain values). So the leading white space is not something unique to dbpa. However, when I go to iTunes, and open any of these mp3 files, iTunes is properly reading the iTunNorm information and showing the soundcheck volume modification correctly (when I look at file info within itunes). So there is no need to remove the leading white space iTunes is reading the info correctly.

Not sure why you are seeing the soundcheck info in two places. iTunNorm is in fact a COMMENT tag, and the file can have multiple comment tags I think. So perhaps it is just a function of the program you are using to view tags. Are you seeing both iTunNorm and that info in Comment tag when you use dbpa to look at file and tag info?

[Spoon]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

The encoder quit? so you do not see the process on the screen?

[AudibleImagery]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

When doing album replaygain, the album gain tags are written after the last track, if your files are on a network share and doing 75K files it could take a very long time to finish.

The files are on my local drive. Everything seemed to complete just fine since I didn't find any empty ITUNNORM/COMMENT ITUNNORM fields. This is the first time the encoding hasn't shown a Results screen (normally it does after a few hours of waiting). If the encoding didn't show a results screen could that be problematic or is that just a minor bug?

[AudibleImagery]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

That's it: Soundcheck. I use foobar2000 MOBILE app on my iphone/ipad as a player (it can use the RG tags). Then I can load up files (even FLAC files) on these devises using TuneFusion (another dbpoweramp product). I use it to do conversions from FLAC to mp3 on the fly for just albums I'm loading up on my iDevices.

I'm jealous of Windows users because Foobar2000 is exactly what I'm looking for. The app for iOS looks promising though and so I'm going to give it a try. Do you know if it supports rating and 'loving' (iTunes/iPhone feature)?

But, the SOUNDCHECK approach *should* work for you. Are you saying it doesn't work? Are you sure you are seeing these tags being written to your files (look at file properties. The soundcheck tag is something like ITUNNORM XXXX, or COMMENT ITUNNORM. That has the soundcheck info.

Soundcheck with dBpoweramp's ReplayGain encoding works most of the time but I've noticed that occasionally there are some tracks that are noticeably quieter or louder than the average track. The tags are being written to ITUNNORM XXXX and COMMENT ITUNNORM but as I explained to Spoon the encoding process didn't display a Results screen and all of the ITUNNORM/COMMENT ITUNNORM tags have leading whitespaces. I found a way to remove the whitespace with Jaikoz (haven't figured out how to perform this function with dBpoweramp or PerfectTunes yet) and as a result the volume normalization seems to be effective. For some reason the ITUNNORM XXXX and COMMENT ITUNNORM tags are replicated in the Comments section in iTunes, do you think I can safely delete them without affecting the ReplayGain?

[garym]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

When doing album replaygain, the album gain tags are written after the last track, if your files are on a network share and doing 75K files it could take a very long time to finish.

try running in smaller batches.....A thru D, E thru H, etc.

[Spoon]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

I am attempting to apply ReplayGain encoding to 75,000 songs (usually played in iTunes) in dBpoweramp's batch converter using the [ReplayGain] encoding. I'm using Track, Album Gain & iTunes Track Normalization with Albums Identified By as "All Files in Same Folder", "ReplayGain" as the Gain Calculation, Maximum Gain at 25 dB (although I'm unsure about this setting), "Disable Clip Prevention" unchecked, and "Preserve Data Modified Time" checked.

In addition to the "dBpoweramp unexpectedly quit error", the encoding also didn't finish. It got to the last song and just was stuck there and never showed me the Results screen.

When doing album replaygain, the album gain tags are written after the last track, if your files are on a network share and doing 75K files it could take a very long time to finish.

[garym]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

I use SoundCheck on both of my devices. Is that what you're referring to?

Do you have any recommendations for media players other than iTunes that could utilize these ReplayGain tags?

That's it: Soundcheck. I use foobar2000 MOBILE app on my iphone/ipad as a player (it can use the RG tags). Then I can load up files (even FLAC files) on these devises using TuneFusion (another dbpoweramp product). I use it to do conversions from FLAC to mp3 on the fly for just albums I'm loading up on my iDevices.

But, the SOUNDCHECK approach *should* work for you. Are you saying it doesn't work? Are you sure you are seeing these tags being written to your files (look at file properties. The soundcheck tag is something like ITUNNORM XXXX, or COMMENT ITUNNORM. That has the soundcheck info.

also, for a player on a windows machine, regular foobar2000 is a great player.

[AudibleImagery]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

those settings are good. Mine is set at maximum of 25db as well. That must be the default. Keep in mind that iTunes and iPhones/iPads don't use the replaygain track or album tags in playback. But they use the iTunes track normalization info to "normalize" playback volume. I haven't used iTunes in a while, but on my apple players, I recall that I always had to turn "on" the volume normalization in the players. That is, the apple devices (ipods for example) didn't use the information unless I turned on a setting telling the iPod to use it. Have you done this?

I use SoundCheck on both of my devices. Is that what you're referring to?

Do you have any recommendations for media players other than iTunes that could utilize these ReplayGain tags?

[garym]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

those settings are good. Mine is set at maximum of 25db as well. That must be the default. Keep in mind that iTunes and iPhones/iPads don't use the replaygain track or album tags in playback. But they use the iTunes track normalization info to "normalize" playback volume. I haven't used iTunes in a while, but on my apple players, I recall that I always had to turn "on" the volume normalization in the players. That is, the apple devices (ipods for example) didn't use the information unless I turned on a setting telling the iPod to use it. Have you done this?

[AudibleImagery]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

Where are you seeing a setting for "Maximum Gain" in the ReplayGain settings? There is no need to "manipulate" replaygain. Simply use the ReplayGain DSP in ripping or converting, and set it to add both ALBUM and TRACK ReplayGain tag info. The default volume level is based on LUFS = -18. This can be set higher or lower in the ReplayGain advanced settings. I use the default of -18 personally.

Where are you seeing a setting for "Maximum Gain" in the ReplayGain settings? There is no need to "manipulate" replaygain. Simply use the ReplayGain DSP in ripping or converting, and set it to add both ALBUM and TRACK ReplayGain tag info. The default volume level is based on LUFS = -18. This can be set higher or lower in the ReplayGain advanced settings. I use the default of -18 personally.

Hey garym, you seem really knowledgeable on this subject based on the countless posts about this topic that I've seen you weigh-in on and I really appreciate all of the knowledge that I've gleaned from you over the years.

By manipulation I just meant "operate" but I haven't been able to get the ReplayGain encoding to cooperate. I play my music in iTunes and I almost exclusively shuffle it. I've set the ReplayGain with Track, Album Gain & iTunes Track Normalization, "All Files in Same Folder", "ReplayGain" as the Gain Calculation, Maximum Gain at 25 dB (although I'm unsure about this setting; please see attached image), "Disable Clip Prevention" unchecked, and "Preserve Data Modified Time" checked.

[AudibleImagery]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

What are you doing exactly?

I am attempting to apply ReplayGain encoding to 75,000 songs (usually played in iTunes) in dBpoweramp's batch converter using the [ReplayGain] encoding. I'm using Track, Album Gain & iTunes Track Normalization with Albums Identified By as "All Files in Same Folder", "ReplayGain" as the Gain Calculation, Maximum Gain at 25 dB (although I'm unsure about this setting), "Disable Clip Prevention" unchecked, and "Preserve Data Modified Time" checked.

In addition to the "dBpoweramp unexpectedly quit error", the encoding also didn't finish. It got to the last song and just was stuck there and never showed me the Results screen.

[garym]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

Thank you for your help so far. I updated to 17.2 and so I'll try to reencode the files and let you know the results.

I'm new to ReplayGain and how to manipulate it. What should I put for "Maximum Gain" if I'm trying to get all of my songs to play at the same volume level?

Where are you seeing a setting for "Maximum Gain" in the ReplayGain settings? There is no need to "manipulate" replaygain. Simply use the ReplayGain DSP in ripping or converting, and set it to add both ALBUM and TRACK ReplayGain tag info. The default volume level is based on LUFS = -18. This can be set higher or lower in the ReplayGain advanced settings. I use the default of -18 personally.

[Spoon]

Re: Mass Removing Leading and Trailing Whitespaces From a Tag Field

What are you doing exactly?