Topics not related to SDCompo.

xrns recovery

xrns recovery

Postby gilli » Sat May 28, 2011 7:58 pm

Just after I had started a song for the nsf compo I accidently delted my whole music directory where bought mp3's and all my modules laid in.

I must have accidently moved my music folder to the trash bin and then I decided to empty the trash. My last backup was three months old. Well done.

So I investigated on what I would have to do to recover my files on an ext3 system.
In case this is going to be any help for any other victim (I really hope there is none), I write down the things needed to recover your files from an affected partition.

If you should have come across this article by accident and not because you actually lost files, then consider the first point, else just skip it:

1) Chances are bad on an ext3 (and ext4) system. Better keep your backups update.

2) You really need an additional partition / harddisk that is bigger than the partition where you lost data from.

3) If you're reading this because you lost data, stop working on that partition immediately. The more your system writes on it, the more deleted files will be overwritten by new files.

4) Boot your system from a stick (an ubuntu or knoppix image). We'll use linux for the recovery.

Now most linux live images provide a tool called "dd" creates images of partitions or extracts them to one. We will use this tool to copy the partition with the lost files to a single image file which will lay on a different partition (note item 2).

5) Become root on the booted linux ("sudo -s" on ubuntu").

6) Do not mount the drive with the lost data.. Again, consider item 2.

6) Now list the available partitions on your system. To do this, I can suggest two ways.
Either on the shell:
Code: Select all
sfdisk -l

Or using gparted. The advantage of gparted is, it will visualize the devices, their partitions and the partition names without need to mount them. This way you can detect your partitions more easily (more specifitally: the one you want to create an image of).

7) The partition where you want to copy your image to has to be mounted now.

8) Now we use dd to create an image of the affected partition.
Be careful using dd. Make sure you don't mix up its source and its target. This tool has no mercy after execution. If it has something to write on, it'll do this without a warning.
Assuming your affected image is "/dev/sda1" and your additional drive is mounted in "/media/backup/" then dd is to be called like this:
Code: Select all
dd if=/dev/sda1 of=/media/backup/image_with_lost_files.dd& pid=$!


9) Now be patient. Depending on the size of your partition you'll need to wait some time until the image is created. You can check occasionally how far the process got by typing:
Code: Select all
kill -USR1 $pid; sleep 1; kill $pid

At the end you've got a file, e.g. "image_with_lost_files.dd".

10) If your desktop os is linux, you can boot into that now as its image has been copied and won't be affected anymore. If your desktop os is something different, you should stay in the booted live distro and hope some of the mentioned recovery tools are provided. We'll use three different linux tools to recover what's possible.

11) If your file system is something like ext3 or ext4, let's suppose the state of your deleted files is still very healthy. So we'll use extundelete which tries to recover file names and is even capable of recovering the files whose path you specify.
Code: Select all
extundelete --restore-file home/music image_with_lost_files.dd

This command tries to recover the specified directory from image. If nothing is found, you'll end up having a directory "RECOVERED_FILES" that contains nothing. So you can try to recover everything it finds:
Code: Select all
extundelete --restore-all home/music image_with_lost_files.dd

Now look in the subdirectory "RECOVERED_FILES" if you got back what you were missing.

12) We all know that Renoise modules are zip archives basically containing the song as an XML file and the samples/instruments in subdirectories. The most promising tool to recover Renoise modules is foremost. It can recognize files by carving, so if it finds a zip file header in the image, it might be one of your renoise modules.
Code: Select all
foremost -i image_with_lost_files.dd -t zip

This command will issue foremost to recover only zip files from the partition image. Once done you'll find a subdirectory called "zip" containing lots of zip files with arbitrary file names. I'll describe below how to recognize Renoise modules in zip files.

If you want foremost to restore every file type it knows (those are limited), you can run it like this:
Code: Select all
foremost -i image_with_lost_files.dd

For each file type it knows, a subdirectory will be created. You'll be able to open the files then to see if they are something you needed.

13) If you want your mp3 files to be restored, too, you'll need PhotoRec. It will do the sames as foremost but is also able to find mp3 files (among other file types). Adittionaly, if it can restore the original file name, it will do so. But it will not recognize zip files. Run it like this:
Code: Select all
photorec image_with_lost_files.dd

Have a look on their homepage to learn to step through the default options to choose on the menu. Once done, PhotoRec will create many subdirectories called something like "recup_dir.1" with a raising number. Now if it's just the mp3's you've been looking for, then you just need to search for the file type and see if you were lucky.

14) Detecting Renoise modules from restored zip files
Renoise modules always contain a "Song.xml" inside the XRNS file (which is a .zip file). Thus we only need a tool that searches in files for a certain pattern, that is "Song.xml". To do this, there is the linux user's all time friend "grep". It's always provided with the coreutils so you can be sure you have it, no matter what distro you're on. Now since we want to have the other zip files being sorted out from the Renoise modules, we'll also copy each file that has a "Song.xml" inside to a different directory. First create a directory for the found Renoise modules. Something like "rescued_songs". Next be sure you're inside the directory containing the recovered zip files (by foremost) and then run:
Code: Select all
find . -maxdepth 1 -type f -exec grep -q "Song.xml" '{}' ';' -exec cp '{}' rescued_modules/ ';'

Take care of the uppercase in "Song.xml". Grep searches case sensitive and we know how the xml file is called. If you wanted to ignore the case, grep would take even longer.

Now you need to be even more patient. Your PC is not only finding files, it's opening them, looking through them and trying to find a string that matches "Song.xml". Each successful find will be copied to the "rescued_modules" directory. Then you'll have a limited amount of files to open and recognize.

15) If you're searching for a specific song and can think of a sample name that's inside it, then you can make grep look for it, too. As file names are not compressed inside a usual zip archive, chances are high you'll find something:
Code: Select all
grep -i "amen" . -R

Now grep will ignore the case and look inside each zip file for the word "amen". If it finds one, it'll tell you something like
Code: Select all
Match found in binary file: ./123456.zip


16) If your only chance is the song title or any other word/sentence that might be written inside the "Song.xml", then you cannot use grep to find it. The file name "Song.xml" is saved as plain text inside a zip file but its content is compressed. That's when zipgrep might help you.
Code: Select all
find . -maxdepth 1 -type f -exec zipgrep -q "www.sdcompo.com" '{}' ';' -exec cp '{}' rescued_modules/ ';'

Now here we're doing the same as in step 14 but this time we're looking inside each zip file to find "www.sdcompo.com" which might be standing in the song comments. It's clear that this step takes a lot of time as you're opening, unzipping and searching through each file now.

17) Restore corrupteed zip files
Many of the recovered zip files might be corrupted as foremost couldn't exactly recreate them from your affected image. In this case you can use a simple command to try to fix them:
Code: Select all
zip -FF 54397904.zip --out somemodule.zip

So this command takes a zip file and saves its fixed version to the file "somemodule.zip".


That's it. I hope you'll never need to go through this but if you should, the examples should help you to know where to start and what can be done.

For me doing these steps restored many files (that I didn't need as I still had them on my old backup) from my ext3 partition but the very last module I had written, which I was searching for, is gone. However, it's all not too bad because I still have different versions of it but at least I could show you how the chances are for getting back deleted files/directories.
User avatar
gilli
Regular
Regular
 
Posts: 47
Joined: Sat May 24, 2008 8:08 am
Location: Germany

Re: xrns recovery

Postby fbjon » Wed Jun 01, 2011 9:14 am

gilli wrote:
Code: Select all
extundelete --restore-file home/music image_with_lost_files.dd

foremost -i image_with_lost_files.dd -t zip

photorec image_with_lost_files.dd

find . -maxdepth 1 -type f -exec zipgrep -q "www.sdcompo.com" '{}' ';' -exec cp '{}' rescued_modules/ ';'


Extundelete! Good lord, finally a way to get stuff back. Also, I noticed there's TestDisk, in the same package as photorec, a tool for recovering lost partitions. Good permanent additions to my toolchest all round, thanks!
_
Image Chotoro gallery
User avatar
fbjon
Regular
Regular
 
Posts: 97
Joined: Thu Apr 06, 2006 3:07 pm

Postby Airmann » Thu Jun 02, 2011 6:00 pm

Wow Gilli,

nice recommendations. Really interesting.
It's really hard to rescue something from ext3

I had a similar case with an accidental rm -rf *.
Airmann
Insomniac
Insomniac
 
Posts: 159
Joined: Sun Aug 31, 2008 4:51 pm
Location: Germany

Postby organic io » Thu Jun 02, 2011 11:27 pm

Airmann wrote:I had a similar case with an accidental rm -rf *.


Matthias.... Dude............... How do you accidentally type that???? :D :D :D :D

I guess there was supposed to be something after the "." but you hit enter too soon?

Please do tell an entertaining story for us :)
User avatar
organic io
Compo Admin
Compo Admin
 
Posts: 1535
Joined: Thu Apr 06, 2006 11:55 am

Postby Airmann » Fri Jun 03, 2011 7:49 pm

^^... ok, easy

I'm working a lot with linux and often I have to remove a bunch of folders. I don't like the interactive question-stuff, so I disabled it (well, one shouldn't actually do that, but...)

On a funny day I typed in a lightspeed quick rm -rf * on / and hit lightspeed-quick return. It was a typo which destroyed the whole server.
No chance undeleting this ! But fortunately, I had a complete backup.

So, I learned from that:
don't work faster than you can think :-)
Airmann
Insomniac
Insomniac
 
Posts: 159
Joined: Sun Aug 31, 2008 4:51 pm
Location: Germany

Return to Off-topic

Who is online

Users browsing this forum: No registered users and 1 guest