Corrupted Flac file - please advise: causes, testing?

[dvdr]

Hi

I don't know, whether this is related to Asset, my computer, my drives or whatever - but please advise what to look for, possible reasons and a way to test my 20.000+ files (mostly flac): I found a corrupted file and am fearing, this might not be the only one...

All over sudden, on one track, playback stopped after 49 seconds. Investigating this (doing a conversion flac -> flac), dbpoweramp said: streaming error in the flac file. Obviously, the file was corrupted somehow.
Luckily, I keep a backup on an external harddisc. The strange thing was: both the corrupted and the backed up file had the same size and date-stamps, but the backed up file was not corrupted. As far as I remember, this file (one of my favourite songs, played fine the last times, heard it quite often during the last weeks) was never touched, moved, converted again after I had ripped it and put it on the Raid-discs. The backup had happened about 4 weeks ago, so in between, the corruption must have happened.

I am running asset on a win xp machine - this machine is the dedicated "audio-server", it's just the server running Asset, no conversions, ripping etc. take place on it, files reside on a Raid 5 system using 3x1 Terabyte Samsung drives, that seem healthy (temp: about 40° Celsius). Lately, I was experimenting with different versions of Asset and reindexing the collection several times from scratch.

So, please help me out:
- what could likely cause file corruptions in the flac stream (the file itself was accessable, so I guess, a windows file error or similar is out of question?)
- what must I do to prevent this
- is there any manageable way to check the other 20.000+ files for a possible file corruption without putting them into danger of getting corrupted as well? Files are mostly flac, some are mp3, wma and mpg-audio. I'd like to do such a check to find out in time, before I back up the files again and run the risk to overwrite ok files on the backup disc....

Thanks for helping!

[Spoon]

Re: Corrupted Flac file - please advise: causes, testing?

dBpoweramp Batch Converter >> Select the Music Folder (or the root of where you audio files are kept) >> Convert To >> Test Conversion

This will highlight corrupted files.

[dvdr]

Re: Corrupted Flac file - please advise: causes, testing?

Thanks, I'll do that.

What are possible reasons for a file to get corrupted inside a system, where files are "just" stored and played back? RAID-problem? Anything during playback?

[Spoon]

Re: Corrupted Flac file - please advise: causes, testing?

What type of Raid?

[dvdr]

Re: Corrupted Flac file - please advise: causes, testing?

Hardware Raid Controller, RAID 5, 3x1TB Samsung discs -> 2TB storage space, no errors reported by the controller.

[ajs6000]

Re: Corrupted Flac file - please advise: causes, testing?

Hi,

I use "audiotester" to check the integrity of my FLAC library. I also have a 2TB raid 5 config, but found that some of my FLAC files had been corrupted during the copy process between my workstation and the raid server.

Audiotester.exe can be found at:

http://www.vuplayer.com/other.php

[dvdr]

Re: Corrupted Flac file - please advise: causes, testing?

Thanks for pointing out the testing options.
I ran a test of half my files yesterday evening and discovered 2 corrupt files: "CRC error" (let's hope for the other half of the files to be ok...). Unfortunately, these files are also corrupted on the external backup drive, that I still keep :-(

Spoon: I know there is an option in dbpoweramp "decode through errors". Could I at least play flac-files (NATIVELY, without forced wav) "through errors" in Asset as well, or would that require the "forced wav" option?

Question to all: would only a copy process bear the danger of file corruption or could that also happen, when a file is modified (filename, rearranging location inside the folder structure, tag editing, reindexing)? I don't 100% exactly know about the mirroring process of a RAID5, but since it's not an exact 1:1 mirror - when would file-actions bear the danger of the file being somehow "copied" (physically moved) inside the RAID5 and that way getting corrupted?

I don't want to sound paranoid, but I have quite valuable files stored there of own recordings. That's why I am keeping an external backup-drive as well. But all backup routines seem to be worthless, if - as it happened - file corruption comes creeping in unnoticed on the backup drive as well. Any suggestions?

[Spoon]

Re: Corrupted Flac file - please advise: causes, testing?

dBpoweramp Configuration >> Codecs Advanced

There is an option to decode flac through the error, enable this, then convert the corrupted flac files, then remove that option so it will allow dbpoweramp to detect other corrupt files in future.

[dvdr]

Re: Corrupted Flac file - please advise: causes, testing?

Thanks Spoon

done that, "converted" the corrupted files to Flac again, unfortunately, there then is a short dropout in the new file - I guess, that's not to avoid, but at least I can use the files again!

[ajs6000]

Re: Corrupted Flac file - please advise: causes, testing?

Hi dvdr
Being paranoid (about data loss or file corruption) is a key element in building a successful backup strategy!

A Raid configuration cannot protect you against file corruption. Unfortunately file corruption can be generated by many factors (faulty software and hardware), including the raid controller that's supposed to be looking after your precious data. Raid is mostly about providing large, fast and highly available data storage.

Therefore your backup strategy cannot make any assumptions about the current reliability of the components storing your data or making the backup and should contain some element of verification to ensure that both the data being saved and the saved data is error free.

Lecture over!

I run my complete library (15,000 files) through "audiotest" before a backup and then run the complete backup through "audiotest". This type of strategy ensures I regularly (every 2 months) know the state of my flac library and can quickly respond to any flac errors. Of course, since running this strategy (2 years) I have not had any errors! But at least I know.

Of course, on a paranoid note, how do you know that “audiotest” is picking up all data corruptions?......

[dvdr]

Re: Corrupted Flac file - please advise: causes, testing?

Hi ajs6000

thanks for pointing out a few key-paranoical-elements of backing up data
I will change my backup strategie and add another element: testing of files put on the RAID and the backup drive.
Do you know off hand, when exactly flacs already existing (error-free) on my HD are endangered to file corruption? Is a simple tag-change already problematic to corrupt the "audio-part" - in other words: after what operations would you consider a file-check necessary?
btw.: the errors, that I encountered in some files were "(FRAME_CRC_MISMATCH @ 0m 15s)", "(LOST SYNC @ END OF FILE)", "(TRUNCATED @ 0m 17s)" and just "(Truncated)" without a time-value. Does that point to possible reasons for the corruption?

Thanks for your help, again.

[ajs6000]

Re: Corrupted Flac file - please advise: causes, testing?

Hi dvdr

With my experience of flac corruption I was able to reproduce the error. The corruption was taking place when I copied the files from a workstation (the master copy) to the media server. The corruption was only occurring randomly so not all files were being corrupted. I wasn't able to directly identify the faulty component. But fixed the problem by reinstalling the server OS (Windows 2003 server). I can only assume that some part of the OS (file management) had itself been corrupted and therefore was not always handling the files correctly.

I don't think you should worry too much about what type of write operation could corrupt a flac file or indeed any part of the file system. However, if you experience any form of system or process halt (program freeze/crash etc) following or during a file write (any form of file modification) then it would be advisable to check these files on system restart.

In general, file corruption is very rare on a well maintained system using either NTFS on Windows or EXT3 on Unix/Linux.

[dvdr]

Re: Corrupted Flac file - please advise: causes, testing?

I don't think you should worry too much.....In general, file corruption is very rare on a well maintained system using either NTFS on Windows or EXT3 on Unix/Linux.

Shoot, now I have to find something else to outlive my paranoia In other words: thanks a lot for helping out with your profound answers!

[garym]

Re: Corrupted Flac file - please advise: causes, testing?

Would converting to "Test Conversion" before and after backing up to a separate disk be a good confirmation that the files are not corrupted? Seems like this would confirm that the file can be read and converted (without actually rewriting the file or converting to anything). Or is there a more efficient/effective way?

[garym]

Re: Corrupted Flac file - please advise: causes, testing?

oops. A little searching and I answered my own question (I think). Yes, the test conversion would check things out for FLAC files. Doesn't seem to work for mp3 files however. Or am I wrong about this. A couple of mp3 files tested as bad by audiotester (and that won't play in foobar), I can run through the TEST CONVERSION and it takes forever, but just ends eventually without any notice of error or information (and I have "show information at end" option selected in TEST CONVERSION.)

Would converting to "Test Conversion" before and after backing up to a separate disk be a good confirmation that the files are not corrupted? Seems like this would confirm that the file can be read and converted (without actually rewriting the file or converting to anything). Or is there a more efficient/effective way?