title
Products            Buy            Support Forum            Professional            About            Codec Central
 
Results 1 to 10 of 10

Thread: Write Metadata File tags _cdgap, _cdindex

  1. #1

    Join Date
    Jan 2010
    Posts
    9

    Write Metadata File tags _cdgap, _cdindex

    I've got R14 reference. I rip to FLAC using individual track files. I'd like to create an external DSP effect that will create a CUE sheet that includes the gap information.

    I turned on the "Write Metadata File" DSP effect in CD Ripper, and saw that the ConvertedFile/IDTags section included two interesting entries, _cdgap and _cdindex, that look to have just the info I need.

    Unfortunately, there's a bug in that DSP effect that makes it useless (already reported by someone in thread 22029).

    So is there some other way to get at those _cdgap and _cdindex tags? They don't seem to be true tags (I don't see them in the FLAC output, using metaflac.exe).

    I tried running the Write Metadata File effect from the batch converter instead, just to see if I still saw those tags. If yes, that would have indicated they were available somehow in the FLAC. But this time, I got a crash in the converter process instead, with an offer to send the crash dump info on to Microsoft. Oh well.

  2. #2
    Administrator
    Join Date
    Apr 2002
    Posts
    38,582

    Re: Write Metadata File tags _cdgap, _cdindex

    Any tags with '_' are for internal use only and would not be written to an end file.

    You might be able to use the DSP effect ID Tag Processing and move those tags to normal tags so they are written.

  3. #3

    Join Date
    Jan 2010
    Posts
    9

    Re: Write Metadata File tags _cdgap, _cdindex

    Thanks, ID Tag Processing works great. Only glitch I encountered was that you need to have the Gap and Index Position columns turned on to get these psuedo-tags populated.

  4. #4
    Administrator
    Join Date
    Apr 2002
    Posts
    38,582

    Re: Write Metadata File tags _cdgap, _cdindex

    That is correct otherwise they are not read from the CD.

  5. #5

    Join Date
    Dec 2010
    Posts
    3

    Re: Write Metadata File tags _cdgap, _cdindex

    Philiplu and Spoon

    Interesting read.

    If I catch your drift, Philiplu, you're trying to make a script/DSP which would allow you to 1. rip to single flac files applying the metadata DSP, and then 2. use a script to extract gaps & indexes to construct a cue sheet for backup purposes?

    That would be the holy grail for many, achieving both bit perfect backup options allowing eg AR verification, and the convenience and superior tagging options of single flac files.

    Have I misunderstood and if not, would you share your thoughts?

    And, thanks for a brilliant prog, Spoon

  6. #6

    Join Date
    Dec 2010
    Posts
    3

    Re: Write Metadata File tags _cdgap, _cdindex

    Double posted...
    Last edited by map; 12-09-2010 at 02:22 PM.

  7. #7

    Join Date
    Jan 2010
    Posts
    9

    Re: Write Metadata File tags _cdgap, _cdindex

    That's not quite what I'm doing. I'm slowly working on ripping around 1500 CDs, for the 2nd time (my wife ripped everything to low-bitrate MP3s a few years ago, when disk space was a consideration). This time around, I want to make sure we never have to do this again. So I want to get as close as I can to being able to recreate a bit perfect copy, which means I want the CD gap/index info. But right now, I'm not so interested in creating cue sheets and single flac files. I just want to make sure I'm not cut off from doing that in the future. As long as I've got the index info, I can always create those at a later date, while using per-track files for now. I delved into FLAC file formats and CD subchannel minutia enough to be confident I can put together whatever I need at a later date.

    So I'm ripping with two DSPs. First, the recommended ReplayGain with track and album gain. Second, the ID Tag Processing DSP, with two maps: _cdgap to CDGap and _cdindex to CDIndex. That gets the gap/index data into my FLACs in a redundant fashion (you can probably recreate either from the other). Those are nonstandard tags, of course, but that's fine for my purposes. I'm also turning on every possible tag, and making sure to save the information log. Also, as mentioned previously, some tags only show up if you turn on the matching columns. So I make sure the "Gap (Pre-Track)", "Index Positions", and "Track Technical" columns are displayed.

    I'm not capturing all the CD subchannel info, like the CDText stuff, since that doesn't seem to be available right now, but the index info is probably all I'm really concerned with.

    One other thing I'm doing is running everything I rip through a Powershell script I wrote that uses metaflac.exe to read all the tags and put them through a sanity check. I'm including that here, though it's not at all commented. Replace *blooper* with the number sign/pound sign/octothorpe/hash in the two places it appears:

    Code:
    param (
        [IO.DirectoryInfo] $root,
        [switch] $nomissing,
        [switch] $fixgaps,
        [switch] $whatif
    )
    
    [Environment]::CurrentDirectory=(Get-Location -PSProvider FileSystem).ProviderPath
    
    $metaflac = "C:\Tools\flac-1.2.1\bin\metaflac.exe"
    
    $requiredTags = @(
        "AccurateRipDiscID",
        "AccurateRipResult",
        "Album",
        "AlbumArtist",
        "Artist",
        "CDDB Disc ID",
        "CDGap",
        "CDIndex",
        "CDTOC",
        "Composer",
        "CRC",
        "Date",
        "DiscNumber",
        "Encoded By",
        "Encoder",
        "Encoder Settings",
        "Genre",
        "Length",
        "Organization",
        "Profile",
        "replaygain_album_gain",
        "replaygain_album_peak",
        "replaygain_track_gain",
        "replaygain_track_peak",
        "Source",
        "Title",
        "TotalDiscs",
        "TotalTracks",
        "TrackNumber",
        "UPC"
    )
    
    $requiredClassicalTags = @(
        "Period"
    )
    
    $optionalTags = @(
        "Album Artist Sort",
        "Artist Sort",
        "Comment",
        "Compilation",
        "ComposerSort",
        "Conductor",
        "ConductorSort",
        "Description",
        "ISRC",
        "Performer",
        "Rating",
        "Style"
    )
    
    $shouldBeIdenticalTags = @(
        "Album",
        "Album Artist Sort",
        "AlbumArtist",
        "CDDB Disc ID",
        "CDTOC",
        "Compilation",
        "Date",
        "DiscNumber",
        "Encoded By",
        "Encoder",
        "Encoder Settings",
        "Genre",
        "Organization",
        "Profile",
        "replaygain_album_gain",
        "replaygain_album_peak",
        "Source",
        "Style",
        "TotalDiscs",
        "TotalTracks",
        "UPC"
    )
    
    $shouldBeDifferentTags = @(
        "AccurateRipDiscID",
        "CDGap",
        "CDIndex",
        "ISRC"
    )
    
    function Assert([scriptblock]$expr)
    {
        if (! (&$expr)) {
            throw "Assertion Failure at $((get-pscallstack)[1].Location): $([string]$expr)"
        }
    }
    
    Function ReadTags([IO.FileInfo]$path)
    {
        $tags = @{}
        $lastKey = $null
        &$metaflac --export-tags-to=- $path.FullName |
        % {
            $key, $value = $_ -split "="
            if ($value -eq $null) {
                $value = $key
                $key = $lastKey
            } else {
                $lastkey = $key
            }
            if ($tags.ContainsKey($key)) {
                if ($tags[$key] -is [array]) {
                    $tags[$key] += $value
                } else {
                    $tags[$key] = $tags[$key], $value
                }
            } else {
                $tags += @{ $key = $value }
            }
        }
        return $tags
    }
    
    function FindAlbums([IO.DirectoryInfo]$root)
    {
        gci $root -recurse -filter *.flac |
        % { $_.DirectoryName } |
        Get-Unique
    }
    
    function ReadAllTracksTags([IO.DirectoryInfo]$album)
    {
        $allTags = @(@())
        foreach ($track in (gci -literalpath $album -filter *.flac)) {
            $tags = ReadTags $track
            Assert {$tags.ContainsKey("DiscNumber")}
            Assert {$tags.ContainsKey("TrackNumber")}
            $discnum = [int]$tags.DiscNumber
            $tracknum = [int]$tags.TrackNumber
            if ($discnum -ge $allTags.count) {
                $allTags += @(@($null)) * ($discnum - $allTags.count + 1)
                $allTags[$discnum] = @(@{})
            }
            if ($tracknum -ge $allTags[$discnum].count) {
                $allTags[$discnum] += @($null) * ($tracknum - $allTags[$discnum].count + 1)
            } elseif ($allTags[$discnum][$tracknum] -ne $null) {
                throw "Two track files found with disc/track *blooper*$discnum/$tracknum in album:`n  $($album.FullName)"
            }
            $allTags[$discnum][$tracknum] = $tags
            $allTags[$discnum][$tracknum].__TRACKFILE__ = $track
            foreach ($key in $tags.Keys) {
                if (!$allTags[$discnum][0].ContainsKey($key)) {
                    $allTags[$discnum][0].$key = $tags.$key
                }
            }
        }
        return $allTags
    }
    
    function CharacterizeTags($trackTags)
    {
        $commonTags = $trackTags[0].Clone()
        for ($i = 1; $i -lt $trackTags.count; ++$i) {
            if ($trackTags[$i] -eq $null) {continue}
            $tempHash = $commonTags.Clone()
            foreach ($key in $tempHash.Keys) {
                if (!$trackTags[$i].ContainsKey($key)) {
                    $commonTags.Remove($key)
                }
            }
        }
        $identicalTags = $commonTags.Clone()
        for ($i = 2; $i -lt $trackTags.count; ++$i) {
            if ($trackTags[$i] -eq $null) {continue}
            $tempHash = $identicalTags.Clone()
            foreach ($key in $tempHash.Keys) {
                if (@(Compare-Object $identicalTags.$key $trackTags[$i].$key -sync 0).length -ne 0) {
                    $identicalTags.Remove($key)
                }
            }
        }
        return $commonTags, $identicalTags
    }
    
    function FindMissingTags($trackTags)
    {
        $missingTags = New-Object object[]($trackTags.count)
        $desiredTags = $requiredTags
        if ($trackTags[1].Genre -eq "Classical") {
            $desiredTags += $requiredClassicalTags
        }
        $commonMissingTags = @{}
        $missingTags[0] = @()
        foreach ($tag in $desiredTags) {
            if (!$trackTags[0].ContainsKey($tag)) {
                $missingTags[0] += $tag
                $commonMissingTags[$tag] = 1
            }
        }
        for ($i = 1; $i -lt $trackTags.count; ++$i) {
            if ($trackTags[$i] -eq $null) {continue}
            $missingTags[$i] = @()
            foreach ($tag in $desiredTags) {
                if (!$commonMissingTags.ContainsKey($tag) -and !$trackTags[$i].ContainsKey($tag)) {
                    $missingTags[$i] += $tag
                }
            }
        }
        return $missingTags
    }
    
    function FormatCommaSeparatedArray([object[]]$array)
    {
        $OFS = ", "
        $text = [string]@($array)
        Remove-Variable OFS
        return $text
    }
    
    function FormatMissingTags($missingTags)
    {
        $text = $null
        for ($i = 0; $i -lt $missingTags.count; ++$i) {
            if ($missingTags[$i] -ne $null) {
                if ($i -eq 0) {
                    $text += "    All tracks: "
                } else {
                    $text += "    Track *blooper*$($i): "
                }
                $text += (FormatCommaSeparatedArray $missingTags[$i]) + "`n"
            }
        }
        if ($text -ne $null) {
            $text = "  Missing Tags:`n" + $text
        }
        return $text
    }
    
    function CheckTagsReasonable($trackTags, $commonTags, $identicalTags)
    {
        $text = $null
        $missingTracks = @()
        for ($i = 1; $i -le $trackTags[0].TotalTracks; ++$i) {
            if ($trackTags[$i] -eq $null) {
                $missingTracks += $i
            }
        }
        if ($missingTracks.count -ne 0) {
            $text += "  Tracks missing (long pathnames?): " + (FormatCommaSeparatedArray $missingTracks) + "`n"
        }
        $mismatchTags = @()
        foreach ($tag in $shouldBeIdenticalTags) {
            if ($trackTags[0].ContainsKey($tag) -and !$identicalTags.ContainsKey($tag)) {
                $mismatchTags += $tag
            }
        }
        if ($mismatchTags.count -ne 0) {
            $text += "  Tags not same across all tracks: " + (FormatCommaSeparatedArray $mismatchTags) + "`n"
        }
        foreach ($tag in $shouldBeDifferentTags) {
            if ($trackTags[0].ContainsKey($tag)) {
                $ok = $true
                $prevValues = @{}
                for ($i = 1; $i -lt $trackTags.count; ++$i) {
                    if ($trackTags[$i] -eq $null) {continue}
                    if ($trackTags[$i].ContainsKey($tag)) {
                        $value = $trackTags[$i].$tag
                        if ($prevValues.ContainsKey($value)) {
                            $ok = $false
                            $prevValues[$value] = @($prevValues[$value]) + $i
                        } else {
                            $prevValues[$value] = $i
                        }
                    }
                }
                if (!$ok) {
                    $text += "  Tag '$tag' duplicated in multiple tracks:`n"
                    foreach ($value in $prevValues.Keys) {
                        if ($prevValues[$value].count -gt 1) {
                            $text += "    '$value' in tracks " + (FormatCommaSeparatedArray $prevValues[$value]) + "`n"
                        }
                    }
                }
            }
        }
        if (!$identicalTags.ContainsKey("Artist")) {
            for ($i = 1; $i -lt $trackTags.count; ++$i) {
                if ($trackTags[$i] -eq $null) {continue}
                if (@($trackTags[$i].Artist) -notcontains $trackTags[0].AlbumArtist) {
                    if ($trackTags[0].Genre -eq "Soundtrack") {
                        if ($trackTags[0].AlbumArtist -ne "Soundtrack") {
                            $text += "  AlbumArtist should be 'Soundtrack', not '$($trackTags[0].AlbumArtist)'`n"
                        }
                    } else {
                        if ($trackTags[0].AlbumArtist -ne "Various Artists") {
                            $text += "  AlbumArtist should be 'Various Artists', not '$($trackTags[0].AlbumArtist)'`n"
                        }
                    }
                }
                break
            }
        }
        if ($trackTags[0].ContainsKey("Compilation")) {
            if ($trackTags[0].Genre -eq "Classical") {
                if ($identicalTags.ContainsKey("Composer")) {
                    $text += "  For classical compilation, Composer should not be '$($identicalTags.Composer)' for all tracks`n"
                }
            } elseif ($trackTags[0].Genre -eq "Soundtrack") {
                if ($trackTags[0].AlbumArtist -ne "Soundtrack") {
                    $text += "  For soundtrack compilation, AlbumArtist should be 'Soundtrack', not '$($trackTags[0].AlbumArtist)'`n"
                }
            } else {
                if ($trackTags[0].AlbumArtist -ne "Various Artists") {
                    $text += "  For this compilation, AlbumArtist should be 'Various Artists', not '$($trackTags[0].AlbumArtist)'`n"
                }
            }
        }
        if ($trackTags[0].ContainsKey("Profile")) {
            if ((($trackTags[0].Genre -eq "Classical") -and ($trackTags[0].Profile -ne "Classical")) `
                -or (($trackTags[0].Genre -ne "Classical") -and ($trackTags[0].Profile -ne "(default)")))
            {
                $text += "  Unexpected profile '$($trackTags[0].Profile)' for genre '$($trackTags[0].Genre)'`n"
            }
        }
        for ($i = 1; $i -le $trackTags.count; ++$i) {
            if ($trackTags[$i] -eq $null) {continue}
            foreach ($tag in $trackTags[$i].keys) {
                if ($tag -eq "Description") {continue}
                if ($trackTags[$i].$tag -is [array]) {
                    $tagArray = $trackTags[$i].$tag
             :label for ($j = 1; $j -lt $tagArray.count; ++$j) {
                        for ($k = 0; $k -lt $j; ++$k) {
                            if ($tagArray[$j] -eq $tagArray[$k]) {
                                $text += "  Track $i has duplicate value in tag '$($tag)': " + (FormatCommaSeparatedArray $tagArray) + "`n"
                                break label
                            }
                        }
                    }
                }
            }
        }
        return $text
    }
    
    function InvokeOrWhatIf([string]$cmd)
    {
        if ($whatif) {
            Write-Host $cmd
        } else {
            Invoke-Expression $cmd
        }
    }
    
    function FixGaps($discTags, $identicalTags)
    {
        $needCDGap = !$discTags[0].ContainsKey("CDGap")
        $needCDIndex = !$discTags[0].ContainsKey("CDIndex")
        if (!$needCDGap -and !$needCDIndex) {
            Write-Host "-fixgaps ignored, CDGap and CDIndex tags seen in at least some tracks"
            return
        }
        Assert {$identicalTags.ContainsKey("CDTOC")}
        Assert {$identicalTags.ContainsKey("TotalTracks")}
        $cdtoc = ($identicalTags["CDTOC"] -split "\+" | % {[int]("0x" + $_)})
        Assert {$cdtoc[0] -eq $identicalTags.TotalTracks}
        for ($i = 1; $i -lt $cdtoc.count - 1; ++$i) {
            $startLBA = $cdtoc[$i] - $cdtoc[1]
            $endLBA = $cdtoc[$i+1] - $cdtoc[1]
            $path = $discTags[$i].__TRACKFILE__.FullName
            if ($needCDGap) {
                InvokeOrWhatIf "&$metaflac `"$path`" `"--set-tag=CDGap=$startLBA`:$endLBA`:$endLBA`""
            }
            if ($needCDIndex) {
                InvokeOrWhatIf "&$metaflac `"$path`" `"--set-tag=CDIndex=$startLBA`:$endLBA`:I1,$startLBA`""
            }
        }
    }
    
    $albums = FindAlbums $root
    if ($albums.count -gt 1 -and $fixgaps) {
        throw "Error: -fixgaps only allowed with single album"
    }
    foreach ($album in $albums) {
        $allTags = ReadAllTracksTags $album
        foreach ($discTags in $allTags[1..($allTags.count-1)]) {
            if ($discTags -eq $null) {continue}
            $commonTags, $identicalTags = CharacterizeTags $discTags
            $disctext = ""
            if ([int]$discTags[0].TotalDiscs -ne 1) {
                $disctext = " (Disc $($discTags[0].DiscNumber))"
            }
            Write-Host "Checking '$($album)$($disctext)'"
            $text = $null
            if (!$nomissing) {
                $missingTags = FindMissingTags $discTags
                $text += FormatMissingTags $missingTags
            }
            $text += CheckTagsReasonable $discTags $commonTags $identicalTags
            Write-Host $text
            if ($fixgaps) {
                FixGaps $discTags $identicalTags
            }
        }
    }
    I was using this script as an opportunity to learn Powershell, so no claims that the code is optimal. The script requires tweaking for someone else's use, at least replacing the location of metaflac.exe. You'd probably have to change the code to reflect your choices on things like naming schemes, as well. The script makes some assumptions about how compilations, multi-CD sets, and soundtracks are ripped. It would probably help to have my naming settings. For non-classical CDs (the default profile), that's:

    Code:
    [IFVALUE]album artist,[album artist],[IFCOMP]Various Artists[][IF!COMP][artist][][]\[album]\[IFMULTI]Disc [disc] - [][track] [title] ([artist])
    and for classical CDs:

    Code:
    Classical\[IFCOMP]Compilations\[album] ([year])[IFVALUE]album artist,\[album artist]\,[][][IF!COMP][composer]\[album] ([year])\[IFVALUE]album artist,[album artist],[artist][][]\[IFMULTI]Disc [disc] - [][track] [title] ([composer] - [artist])

  8. #8

    Join Date
    Dec 2010
    Posts
    3

    Re: Write Metadata File tags _cdgap, _cdindex

    Hi Philiplu

    Many thanks for your extensive reply which will certainly have an impact on my pending ripping scheme.

    I apologize for having been fundamentally unclear in my earlier comment: I'm really, like you, after individual flac files, not single flac files which dbpa already supports. Much better tagging allowed with individual flac files - e.g. track composers which can't be added if you rip to single flac + cue.

    The index and gap data would allow one to go back restore the audio perfectly. I too am not fussed about cdtext - really a bit antiquated as tagging goes anyway..

    So, basically you're doing exactly what I'd like to do as I start on my 600+ cd collection.

    It's all about having a good 'raw' rip to start with that will allow you to restore the original data (noli ponti ignii consumere, as the Romans would have it - currently reading a rather pompous textbook filled with that sort of jargon). If the data is there, surely a script will allow us to manipulate it later on.

    I like your approach with powershell. I'll have to try to find the time to read up on it, but I will probably do pretty much like you. Many thanks for supplying your name settings as well. I'm also doing a broad mix of pop/rock/classical etc.

    Your post would be really helpful to a lot of people. From reading all over this and other forums for ripping schemes, many prefer individual flac files with a cue sheet file for restoration purposes and therefore stick to EAC, but this would allow you essentially the same data (in a different format, but easily transformed) with the much superior tagging and overall ease of use afforded by dbpa. I agree with Spoon, if I read between the lines a bit, that cue sheets are really unnecessary in the vast amount of cases, but many would argue that getting it right in the first place should include the <1% of cases where gaps and indexes might be non standard. If no extra work would be required, I'd have to agree, and you pretty much seem to have got that worked out.

    Much indebted.

  9. #9
    dBpoweramp Enthusiast
    Join Date
    Sep 2006
    Location
    Cambridge, UK
    Posts
    66

    Re: Write Metadata File tags _cdgap, _cdindex

    Quote Originally Posted by philiplu View Post
    I've got R14 reference. I rip to FLAC using individual track files. I'd like to create an external DSP effect that will create a CUE sheet that includes the gap information.

    I turned on the "Write Metadata File" DSP effect in CD Ripper, and saw that the ConvertedFile/IDTags section included two interesting entries, _cdgap and _cdindex, that look to have just the info I need.

    Unfortunately, there's a bug in that DSP effect that makes it useless (already reported by someone in thread 22029).

    So is there some other way to get at those _cdgap and _cdindex tags? They don't seem to be true tags (I don't see them in the FLAC output, using metaflac.exe).

    I tried running the Write Metadata File effect from the batch converter instead, just to see if I still saw those tags. If yes, that would have indicated they were available somehow in the FLAC. But this time, I got a crash in the converter process instead, with an offer to send the crash dump info on to Microsoft. Oh well.
    Sorry to resurrect such an old thread...

    Can i ask, have these _cdgap & _cdindex fields been removed? I've just looked in both the Write Metadata File and ID Tag Processing DSP plugins and neither list them.

    Is there another way in which we can write track gap and index data to files?

  10. #10

    Join Date
    Jan 2010
    Posts
    9

    Re: Write Metadata File tags _cdgap, _cdindex

    Quote Originally Posted by Salem874 View Post
    Sorry to resurrect such an old thread...

    Can i ask, have these _cdgap & _cdindex fields been removed? I've just looked in both the Write Metadata File and ID Tag Processing DSP plugins and neither list them.
    They're still there, as of dbPoweramp 64-bit Reference Release 15.3 that I last used a few months ago, the last time I ripped anything. I'm not surprised if they don't show up in the plugins, since they're prefixed with '_' and thus technically internal tags. But as long as you add the "Gap (Pre-Track)", "Index Positions", and "Track Technical" columns for display, they should show generally show up. Sometimes they don't at first, and I have to refresh the metadata.

    As described earlier, in addition to turning on the display columns so the internal tags appear, I also use the ID Tag Processing DSP to map _cdgap to CDGap and _cdindex to CDIndex.

    Note that I'm not using the powershell script I had 5 1/2 years ago. Instead, I now use a python script which I describe at Python-scripts-to-improve-my-CD-Ripper-mp3tag-workflow. That script, among a lot of other things, makes sure that the cdgap and cdindex tags are present, so I'm warned to refresh and rerip if something went wrong.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •