'True' Archival Project - Input Request from You Geniuses

[Hope I dropped this into the correct forum. ]

Hey all,

This is my first post, so hopefully I don't get flamed too badly for asking for opinions as I am.

I used dBpoweramp several years ago, along with PerfectTUNES, to do some mild work on my personal music library.

Now, and without boring anyone with details, I/we now have a serious start-up dedicated to arts education, and I'm juggling a tremendous amount of materials, tech development, and a million and one business matters. In short, I'm knowledgeable about a lot of things, but I don't always have the time I'd like to spend to thoroughly read-up on current best practices or newer developments than what I used to know. I'm hoping you good folks can let me know what you believe would be the BEST way to accomplish a few necessary tasks.

In this case, I'm looking at the ripping of thousands of CDs, ensuring the highest quality captures of the optical media, while also presumably/preferably keeping exact copies of the discs "in the archive" for any future needs.

Basically, think of this situation as an institution trying to establish best practices and guidelines before jumping into proper archival of assets - in this case CDs.

This is not my area of expertise, but I learn quickly and most importantly want to do the material justice. Without elaborating, much of the content is niche. Ideally we have 2 copies of every disc: 1 used for ripping (ideally once) and one wrapped as redundant back-up. Overkill? Probably. But this is a longterm venture taking something close to academic institution policies.

Key Considerations:
- We are almost entirely Mac-based.
- As if this exact moment, actual usage of the files is still in limbo (who among our research team creating content and answering questions will have access, delivery mechanism(s), etc.) The more options, the better. Future-proof is always a goal.
- In the past, I've opted to use ALAC, simply due to being a hardcore Mac user. Since it would be nice to be able to play the resulting files on as many types of devices as possible, now and in the future, it may be more appropriate to use FLAC.
- Regardless, lossless is naturally imperative.
- A disc image backup of some type seems like best practice to me. That said, if we are speaking about a single-track solution and CUE sheets, I have very little knowledge of the pros/cons or the specifics. Woefully behind in that regard.

- Because of the Arts nature of this project, metadata is a HUGE aspect and consideration. In the past I spent a tremendous amount of time in this area, because I'm quite OCD about the accuracy and depth of the data, and most sources with information *for this area of music* have a fair amount of info that has to be corrected anyway. The problem is that accepted metadata practices pretty much go against common sense for much of THIS type of music. For instance, if we talk about the original broadway cast recording of Hamilton, the common sense method of identification (be it metadata or filenames) would oftentimes make something like the following the most sense:

Artist: Hamilton
Album: 2015 Original Broadway Cast
Tracks labeled with their full titles, and each person identified in liner notes (or the score) listed as track artists for that track. Tracks comprised of multiple songs would try to find a way to designate, in a short way, which track artists are associated with which "song". Ideally that might be a slash or hyphen, but that's more specific to supporting files and not metadata, I suppose.

And without even continuing with that formatting example, you will know that it makes it a real pain because you are battling what the playback software of choice wants to do with metadata. Particularly if using something like Roon (not my favorite solution, and I don't think it would be the best for this situation, anyway, but it's an example.)

So, friends, kindly share how you would handle something with the scope and specifics as described above. Space is not really an issue (primary local copies of files being stored to SSD. Any reason to even discuss whether Mac OS Extended (Journaled) or APFS, or others, is preferable? WIth these file sizes I'm assuming not, but do let me know if I'm not taking something into consideration.) Highest quality and preservation are the most important concerns.

And if anyone cares, or I need to know it is a BAD drive, I have a new ASUS BW-16D1X-U set aside as the primary drive to use. I've got a few other brands/models if needed, but MOST discs SHOULD be free of any surface errors.

Using PerfectTUNES 3.1, dBpoweramp R17.7

Questions, corrections, workflow suggestions... anything anyone can offer would be much appreciated. This is a long and ongoing project, so it is important to me/us to get it right from the start.

Looking forward to reading posts or private messages. Thanks for taking this request seriously, and hopefully not thinking I'm too much of an ID10T problem to be worth sharing your insights.

You guys seriously rock.

[QUOTE=MusicArchivist;212281]Key Considerations:
- We are almost entirely Mac-based.
- A disc image backup of some type seems like best practice to me. That said, if we are speaking about a single-track solution and CUE sheets, I have very little knowledge of the pros/cons or the specifics. Woefully behind in that regard.
[/QUOTE]
Because you mention archival, the first big problem is that I don't think the mac based dBpoweramp creates CUE sheets, which means you won't be able to create bit perfect copies of your discs whilst retaining the CD layout and audio metadata.
You'll lose things like pregap, index (beyond index 1) and pre-emphasis flags (I think dBpoweramp decodes this for you by default which may not be what you want). These are features that were reasonably common in the 80's and 90's but not any more.

Maybe someone using dBpoweramp on a mac can confirm.

Hello! Yes, I believe you are correct. I've seen such discussion in older threads here, and the last update i could find was that it still wasn't really a priority, and it didn't seem like that has changed.

A couple of thoughts/questions.

1. Why do you need cuesheets,single images. I understand that such a format would allow one to recreate the CD with pre-gap, indexes, etc. But those things are only important if one is trying to take the digital files and burn them BACK to a CD that would be identical to the ripped CD (and would presumably pass AccurateRip* if that disk was ripped again). But in most cases (including mine), I NEVER, EVER want to have this music back on a physical CD. With modern ripping to lossless (I use FLAC) your rips retain gapless info and an album plays back just as the CD (and the audio of each track is bit perfect). I can play something like Abbey Road or Dark Side of the Moon and the gapless track transition is retained, even though I've ripped the CD to separate tracks. (I think in the early days of CD ripping, cuesheets were more important as that was the only way to get gapless back then.)

*and note that a ripped CD's separate tracks can still be tested for AccurateRip, without the need for a cuesheet, etc. PerfectTunes can do this easily.

2. Whatever path you take, I'd suggest ripping/tagging a number of different types of CDs (compilations, classical, pop, multidisk, single artists, multiple artist/non-compilations, etc.) and see how various players handle things. Trying to have a 100% perfect plan before starting rarely works. I see that happen all the time here. People rip 100s or more CDs with their perfect plan, and then later on realize in actual use, that they made wrong decisions about metadata, file naming, etc. So some hands on testing of different approaches will give you some feedback on how things work and what you might want to change.

[QUOTE=MusicArchivist;212285]
Key Considerations:
- We are almost entirely Mac-based.
- As if this exact moment, actual usage of the files is still in limbo (who among our research team creating content and answering questions will have access, delivery mechanism(s), etc.) The more options, the better. Future-proof is always a goal.
- In the past, I've opted to use ALAC, simply due to being a hardcore Mac user. Since it would be nice to be able to play the resulting files on as many types of devices as possible, now and in the future, it may be more appropriate to use FLAC.
- Regardless, lossless is naturally imperative.
- A disc image backup of some type seems like best practice to me. That said, if we are speaking about a single-track solution and CUE sheets, I have very little knowledge of the pros/cons or the specifics. Woefully behind in that regard.
[/QUOTE]

I am not expert in the technical details of this, but my understanding is that an [U]exact[/U] 100% replica of a CD is not possible without some form of industry level hardware and knowledge of the individual pressings. Yes, the actual audio can be extracted 100%, but you cannot exactly capture all of the CD layout and strucuture into a digital form. To be honest, I find the below (old!) article more of an interesting read that anything, but I wanted you to be aware of this given the archival nature of your project.

[QUOTE]
[url]https://web.archive.org/web/20120903210117/http://www.digital-inn.de/exact-audio-copy-english/28787-andre-wiethoff-who-feels-have-say-offsets.html[/url]
[/QUOTE]

My next thought is have you considered the amount of time and effort this is going to take to accomplish? For me, building a lossless FLAC collection has very much been an on and off lengthy process, so 300 CDs has taken me over two years (I also spend a lot of time listening to my ripped music!). Here, I am not really talking about the CD ripping process itself as this is relatively quick! It is the consistency and attention to detail with metadata that takes the time, including coming up with your set of rules on what tags you will use and how they will be formatted. You can bet there will be exceptions and gotchas; that require you to re-evaluate the way you are doing things past, present, and future. My experience is that less is often more.

CD Ripper is an excellent program and I thank Spoon for his engineering with AccurateRip over the years. However, I personally use other 3rd-party tools to do most of the other jobs as part of my workflow. I will detail these below.

[B]CUETools:[/B] if a CD is not in the AccurateRip database (a very small percentage these days!) then I will run a Verify operation in CUETools which will almost always provide a set of matches to the CUETools database. I will copy the results into a seperate .TXT file to compliment my log file from CD Ripper.

[B]Exact Audio Copy:[/B] I use this to generate a non-compliant .CUE file which takes less than 30 seconds. As far as I know, this is the only way to generate a .CUE file that contains the CD layout and structure, thus enabling you to produce an exact replica of the original CD (with the caveat of my first comment). CD Ripper will not produce an equivalent of this. As garym has indicated, most people do not need .CUE files, but given the archival nature of your project, it seems you will probably want them. I like them so I can burn a CD for my car without needing the original media on hand. This use case will not last forever though. You will also need to consider how to handle the Enhanced-CD format. I was using [B]DAEMON Tools Pro[/B] to build an .MDX of the entire CD which would encompass all the extra content and the audio (although without the benefit of AccurateRip here). Not sure I will bother continuing with this given the space taken up and general uselessness of content found on Enhanced-CDs (which can also be copy protection stuff).

[B]foobar2000:[/B] I do all my ReplayGain tag work here as it is just easier to manage across multiple discs and albums in situ. An excellent and highly customisable media player too.

[B]Mp3tag:[/B] the mainstay for me in managing metadata for a lossless collection. I do ALL my metadata work here but there is quite a steep learning curve for some of the automation, given this is a powerful program.

One thing you have not mentioned is album art which will take a significant amount of time to do properly. Again, most people are happy with what CD Ripper or PerfectTUNES provide, but I find the quality from a Google Image search less than acceptable in most cases. My solution here is to grab the iTunes high resolution artwork or use Album Art Exchange. If both turn up a blank, I will scan and touch-up the artwork from the CD sleeve. I use [B]ExifTool[/B] to remove any metadata that may be held within image files before embedding into the .FLAC files themselves.

I have a lot of Andean CDs and Japanese CDs, so you can expect to do a lot more research and work depending on the obscurity and popularity of the CDs you have.

Storage is so cheap I would never burn another disk. Three 1T SSDs in a raid array will handle the data with good redundancy. Then you have to consider the zombie apocalypse. Weapons on site, backups off site. The cloud maybe (ie, other peoples servers). So that is storage.

Organization: A CD is a dicreet object and it's content along with metatata etc. creates a data block with known (?)elements. So a database. Thickest book you can find on SQL and rule one is if you try to add a data field after the thing is set up you will regret it. It's an I/O issue. Your archivee is useless unless it can be searched and accessed in a variety of ways in the future. Making a heap and telling people to dig through it is a no-no. On the input end you need raw personpower (sorry, PC attack) to do research and data entry. Vonunteers, interns, retirees wanting a scrap of yor grant money, and so on.

Running the disks throught the ripper + easy, the superstructure = not easy. I am ripping my entire collection at the moment, auduit5ioning most discks and then ripping selected tracks. But I have found as I go along that when I rip an entire disk it is for a particular purpose. And now of course those whole disk rips are spread throughout well over 1000 tracks ripped from individual disks. Given their purpose I need to go through everything and get them all in a folder..

I use dbpoweramp for ripping and Musicbrainz Picard for tag editing. This combo works well on the Mac.