Missing the root.disk

I wrote earlier about the Mystery of the disappearing root.disk. I've seen a number of additional cases since that post - and not all have been catastrophic, so thought I'd discuss what the steps are to attempt recovery.

My belief is that this problem is largely the result of manually shutting down your computer when the Wubi install is hanging. But there have been cases where corruption has occurred without manual poweroffs and I've noticed that some files I delete from within Wubi are 'recovered' later by Windows.

But it is important for users to resist the urge to 'hit the restart/power button' when the Wubi install is hanging. For any OS this is a bad idea and can lead to problems. For Wubi installs it can be fatal. Instead refer to the Wubi Guide for other options.

The first thing you'll notice may be Windows running an automatic CHKDSK or when you try to boot Ubuntu you end up at a GRUB> prompt.
Then when you boot Windows and look in the C:\ubuntu\disks directory you'll notice the root.disk is missing. In some cases, the \ubuntu\disks directory is missing completely or is corrupted.

Running chkdsk
Depending on the problem, Windows may have run an automatic chkdsk or you may need to run it manually. It's not always necessary, but it will never hurt, so the first thing is to run it.
Go to My Computer on XP or Computer on Vista/7, right click the drive you installed Ubuntu on e.g. right click on C:, select Properties, select the Tools tab, then under Error-checking click Check now. Select to Automatically fix file system errors or Scan for and attempt recovery of bad sectors (I usually don't use this, but if you hard drive has problems it's a good idea). When the drive you installed on is C: the computer will tell you it has to schedule the scan for the next time you start your computer. Reboot to complete.

Locate recovered files/directories
The first thing to do is to look for the \found.000 folder on the drive in question i.e. C:\found.0000. This is hidden by default and (on my Windows 7 install) I also had to tell Windows not to "Hide protected OS files" just to see it. You may also have to recover from an administrator command prompt on Win7 (see below).

So now you look for your root.disk (or other .disk files) and copy them back to the \ubuntu\disks folder. If the entire \ubuntu\disks folder is missing, you'll likely find a dir0000.chk directory and within that the root.disk, swap.disk and empty \boot\grub folders. Copy this back to \ubuntu renaming the directory to disks.

If you're missing the root.disk but there is no file of that name, it may have been renamed chk0000.chk. Rename this to root.disk and copy back to \ubuntu\disks.

If the corruption was minor, then likely everything will work fine. If the corruption is major Windows may not even recover the root.disk at all.

Win7/Vista command line instructions
Hit the START key, enter CMD, then look above and right click on CMD.exe and select "Run as Administrator", as shown below. From Windows 8, type "CMD" on the Metro page, right click, and then look on the bottom for "Run as administrator.

Check for the hidden recovery directories: dir /a:h

Check each \found.??? directory:

C:\>cd \found.000
C:\found.000>dir
 Volume in drive C is OS
 Volume Serial Number is B4B7-99A8

 Directory of C:\found.000

19/07/2011  02:02 PM    15,000,000,000 file0000.chk
               1 File(s) 15,000,000,000 bytes
               0 Dir(s)  222,258,069,504 bytes free

C:\found.000>move file0000.chk \ubuntu\disks\root.disk
        1 file(s) moved.

Or if the whole \ubuntu\disks folder is missing:

C:\>cd \found.000

C:\found.000>dir
 Volume in drive C is OS
 Volume Serial Number is B4B7-99A8

 Directory of C:\found.000

19/10/2012  04:51 PM    <DIR>          .
19/10/2012  04:51 PM    <DIR>          ..
19/07/2011  02:02 PM    <DIR>          dir0000.chk
               0 File(s)              0 bytes
               3 Dir(s)  222,258,069,504 bytes free

C:\found.000>dir dir0000.chk
 Volume in drive C is OS
 Volume Serial Number is B4B7-99A8

 Directory of C:\found.000\dir0000.chk

19/10/2012  04:51 PM    <DIR>          .
19/10/2012  04:51 PM    <DIR>          ..
24/02/2012  12:22 AM    <DIR>          boot
06/11/2012  09:28 AM    13,000,000,000 root.disk
15/11/2011  09:28 PM       268,435,456 swap.disk
               2 File(s) 13,268,435,456 bytes
               3 Dir(s)  127,904,968,704 bytes free

C:\>move dir0000.chk \ubuntu\disks
        1 dir(s) moved.

I hope this helps you to recover your files. Remember to backup important data on your Wubi install. There's no reason to install important personal files on a root.disk - you can access them easily on the /host partition.

Updated 2012-11-06: added some more screenshots, enhanced DOS commands

Wubi boot issues a thing of the past

With the release of the third update of Lucid Lynx (10.04.3) last week, this finally puts to bed the problems with Wubi and Grub2 (Ubuntu's bootloader) that resulted in many Wubi boot failures. The issue that allowed the user to overwrite the Windows bootloader is also taken care of.

These issues were fixed prior to the release of Natty Narwhal earlier this year, however, the patches to the active stable releases (Lucid and Maverick) were only released as an Update in June, and finally in 10.04.3 last week.

This was certainly a big problem that has plagued Wubi users for over a year; longer if you include the problem in 9.10 Karmic Koala. It kept many on ubuntuforums.org busy and led to the successful Wubi megathread (thanks to Rubi1200 for that).

I have been quite critical of Canonical and the developers on this blog for not addressing this sooner (this problem was probably the driving force behind the blog in the first place), however, I think it's true that my focus is quite narrow and the developers have many issues to deal with, and Wubi wasn't always on the top of that list.

It appears that now there is a strong development effort to improve Wubi for 11.10 Oneiric Ocelot - which I'll write about later. So far, I've been running a development Oneiric Ocelot Wubi install from early on in the development cycle without any issues.

One thing seems certain - Wubi is popular as ever with new users and a great tool to introduce Windows users to Ubuntu.

Wubi Natty will be ready for Beta2

Wubi has seen some major focus from Canonical's developers (Colin Watson) in the past week or so, finding and fixing major bugs including long-standing Grub-related bugs affecting 10.04 Lucid Lynx and 10.10 Maverick Meerkat (these releases will see the fixes a little later). But for 11.04 Natty Narwhal, Wubi is looking stable now for Beta2 - or if you're using the current-daily CD images.

This means that both new 11.04 installs as well as upgrades from 10.10 to 11.04 are now working!

PS There's just one innocuous message you'll see before the grub menu is displayed, regarding the prefix not being set.

New release of Wubi migration script

I've released a new version of the Wubi migration script (wubi-move-2.0.sh). Here are some of the new features:

• You can migrate either Wubi or a normal install. It detects the type and handles accordingly. This is useful if you want to move your regular Ubuntu install, keep a working backup, or do some experimentation without jeopardizing anything. Also for Wubi users who migrate and then need to re-jig their partitions, it makes it easier to move the install (without worrying about UUID changes).
• It now supports grub-legacy. This was one area that was lacking - I don't know how many people have Wubi with grub-legacy, but there have been a few requests. So the migration supports releases with grub-legacy: 8.04 to 9.04 (as well as those that have been upgraded to 9.10 and later). The caveat is that the script replaces grub-legacy with Grub2 on the migrated install - so you probably don't want to be doing this prior to release 9.10 (I don't know how robust Grub2 is on 8.04.4, for example).
• It also supports migration from a root.disk file with the --root-disk= option. I had already come up with a manual solution for this before and it didn't seem too big a deal to add in to the script. So if you have a good root.disk, but you've lost the ability to boot into it, or you want to migrate to another machine, you can now boot a live CD and migrate from it. The root.disk must be a working, fully-contained Wubi install i.e. not have separate virtual disks for /boot, /usr or /home. (This rules out grub-legacy where /boot is always on the /host Windows partition.)
• Finally there are some minor tweaks. The new option --shared-swap will bypass the 'mkswap' command to preserve the UUID on the swap partition. Useful if you have multiple installs and you want to share the existing swap partition.
• I also split the output from --help into two separate options: --help and --notes to suppress the flood of output.
• Other notable changes - better validation, better error handling, for instance if there is an error in the chroot, the script attempts to cleanly unmount from the chroot before exiting. 

Testing all the possible scenarios and releases is a challenge. I've done a number of cases from releases 8.04.4 to 10.10 (except 8.10) including Wubi and non-Wubi, with Grub2 and Grub-legacy. But as always community feedback is always welcome.

Wubi and grub problem update

It's been one month since the latest grub-pc update in version 10.04.1 Lucid Lynx, started breaking Wubi installs. And that's a good time to reassess the situation.

I'm not sure how I noticed it - maybe I was browsing Wubi bugs that day - but my quick alert to the issues didn't help. Again, I was hoping for a quick response, maybe the update being pulled - I don't know. But apparently launchpad doesn't have a recall mechanism. At the time I thought it would only affect users who didn't install on the same partition as Windows - the usual grub trick that gets users to install the grub bootloader to their drive masterbootrecord MBR caused by a bug in the 'lupin' package (see here, or here).
But quickly it became apparent that it also affected users who installed on the same partition - a problem we already knew about as it was discovered in the Maverick upgrade. But for 10.04.1 the problem is worse, and requires a manual loop mount of the virtual disk and an edit of the grub configuration file from a live CD.

So now we are on the Christmas break, and the developers get a well-needed rest from all the Natty development, but sadly noone has taken the time to patch lupin, for the many existing users on the supported releases. Still, the volunteers on ubuntuforums.org will soldier on, steering Wubi users to the solution... the Wubi megathread. It's interesting to note that since this thread was created 2 weeks ago, it's had over 2600 visits. And that's not including the number of people who had to be helped before it was created.

So... where are we at? There's not much really to be done to warn users unless they happen to post a support request for an unrelated Wubi question. I'm loathe to join the Wubi-bashing crowd  (although it's tempting): there's still a niche market for users that might cause more damage trying to do a proper partitioned Ubuntu installation. In fact, the new Ubiquity installer has it's own issues and has already caught out a few users and overwritten Windows and all other partitions - see bug 659106.
And in any case, despite all the Wubi-bashing, Wubi appears to be growing in popularity, fueled by its new prominent place on the ubuntu.com download page.

So... I guess we just wait it out, and hope that eventually the grub devs will grace us with their attention and provide a fix.

Canonical has a strange sense of priorities

So it's nearly a month since the latest Grub update in the current Long Term Release, Lucid Lynx 10.04, put the boots to Wubi installs. In addition to the usual alarming, but generally harmless, grub rescue> prompt problem,  which has been around since Lucid was first released in April, the newer failure to boot issue has been totally ignored by Canonical.
This boot failure problem started cropping up in July with grub errors about 'loadfont', but didn't seem to affect everyone. With the November 20 release it now appears to be much more widespread - a fresh install of 10.04.1 with the updates applied does not boot. (*Ubuntu must be installed to the same partition as Windows for this problem to occur).

The reason that it only affects Wubi installs on the same partition as Windows is that it is the grub-install command that is corrupting the /boot/grub directory, and when you update package grub-pc it runs grub-install to try to update the wubildr (Wubi bootloader) - believed by the grub devs to be a required action; BUT, Wubi installs on a different partition slip through the net as the buggy code fails to identify them as Wubi installs. This means that grub-install aborts and the wubildr is not updated - but ironically these still boot.
So the whole idea of updating wubildr and breaking Wubi installs in the process is pointless - why don't they just bypass the update altogether.

So... why haven't the devs rushed to fix this? Even the Ubuntu Release Manager Steve Langasek was notified of the 'critical' issue... but no response at all. The devs appear to be hard at work preparing for the next Ubuntu release (11.04 Natty Narwhal) - in the meantime, unsuspecting users who decide to try out Ubuntu are left to discover what a grub rescue prompt is, or why their new OS fails to boot.

So, in summary, Canonical doesn't seem to give a hoot about the current users who are breaking their computers trying out the 'user friendly' Wubi option. Are they just blindly churning out (buggy) release after (buggy) release as though this is a strategy for long-term success? Hard to say, but let's just say - I'm confused. If they don't want to support Wubi, why don't they just pull it from their main download site to protect the computers of unsuspecting users?

And thanks to everyone on Ubuntuforums.org who are working hard to help the affected users. It's great to have such strong community support, but if Canonical is relying on this for current release support it's pretty pathetic.

Where in the world is Ago

Agostino Russo, the person responsible for creating and maintaining Wubi, seems to have disappeared from sight. Since the run-up to the release of Ubuntu 10.04 (in April this year), he has been conspicuous by his absence.

From Ubuntu IRC logs, it seems that Colin Watson is now responsible for Wubi, but is either unable or unwilling to deal with seemingly minor, showstopper-type bugs that Wubi has suffered from lately. In fact, the bugs appear to be from Colin's own code changes in Grub2.

So what is Canonical's strategy on Wubi? If I were a conspiracy theorist I would suggest that the developers wish to have Wubi removed by making it so unstable as to be unusable. But I am not into conspiracy theories, and I instead believe that it's more likely due to lax maintenance/testing standards.

So that brings me to my point. Where is Ago? He's clearly the only one who has been able to keep Wubi going, and hopefully he can find the time to get things back on track.

Wubi Mega-help-thread

There is now a one-stop Wubi 'megathread' on ubuntuforums.org to repair the grub issues Wubi users have been experiencing. Recommended reading if you are using Wubi, even if you haven't had any issues yet.

The thread is: http://ubuntuforums.org/showthread.php?t=1639198

WUBI and GRUB Issues

Edit Dec 16: If you're currently having problems with Wubi booting or get a grub rescue prompt when booting, see this post: Wubi Mega-help-thread

Since version 9.10, Ubuntu has used Grub2 as the default bootloader. (I'll refer to Grub2 as Grub, as the previous version is now referred to as grub-legacy. )

Grub stands for GRand Unified Bootloader. And it's so grand that everyone is talking about it. You shouldn't notice a good bootloader or need to know it's name, but once you've turned on your computer and seen the Grub prompt screen, or the optimistically named Grub rescue prompt (which means you're totally messed), it's hard not to take notice.

Let's look at some of the significant Grub problems with Wubi:

Karmic Koala 9.10
Problem: Select Ubuntu after running system updates and the result is a grub prompt.
Reason: The wubildr (Wubi bootloader) file can't address files that are 4GB or more from the start of the partition.
Fix: Users must manually download a new copy of wubildr.
Note: This bug is part of the release, so every new 9.10 install will be affected by this.

Lucid Lynx 10.04
Problem: Many users render their computer unbootable, destroy their Windows partition bootsectors. In some cases Windows can not be repaired and requires a reinstall. Some users inadvertently end up losing all their data through bad repair attempts.
Reason: Running the upgrade to 10.04 or even the standard grub updates asks you to select a device to install grub. It confusingly lists all drives and partitions in a user unfriendly way (/dev/sda, /dev/sda1, /dev/sda2 etc.) and then cautions to 'select all drives if uncertain'.
Fix: If you got the right help, reinstalling the windows bootloader and restoring the partition bootsector was all you needed to do. This worked for most people, but as I mentioned, some were not so fortunate.
Note: the developers didn't even respond to the bug report for about 2 months following the release, despite hundreds of affected users (documented cases, but probably thousands were affected). Even non-wubi dual boot installs were affected.

Lucid Lynx 10.04.1
Problem 1: Many users render their computer unbootable.
Reason 1: A fix for the 10.04 grub issue has been released. For 10.04.1 you cannot overwrite your drive bootloader with a grub update if you have a wubi install. Problem is, the means of identifying a wubi install is buggy - only those who installed to drive 'C:' (or their main windows partition) are safe.
Fix 1: Reinstall windows bootloader.
Note 1: The bug that prevents grub from identifying a wubi install is still present in release 10.10.

Problem 2: After selecting Ubuntu, you see errors listed and either hang, reboot, or end at a grub prompt.
Reason 2: Grub now has the ability to update the wubildr on the fly. Since grub is a rapidly evolving product, it needs to update the wubildr to be compatible with all the changes. Somewhere between updating the modules, the grub menu and the wubildr file things go bad. Exact cause unknown. Only affects installs capable of updating the wubildr (on the same partition as Windows)
Fix 2: Loop mounting the root.disk from a 'Live CD' and manually editing the /boot/grub/grub.cfg file (just leave the parts you need to boot i.e. anything on or after the first line starting with "menuentry". For some reason, after booting and running "sudo update-grub" the problem disappears. [Edit: a subsequent grub update in November, is not permanently fixed by running update-grub. Manual clean up of the /boot/grub directory is required - see later posts for info]


Maverick Meerkat 10.10
Problem: After upgrading from 10.04.1 to 10.10, your computer either hangs, reboots, or drops you at a grub prompt.
Reason: Same reason as for problem 2 on 10.04.1 - for installs to the same partition as Windows, the wubildr file is updated and corrupted.
Fix: Copying the backup wubildr from the c:\ubuntu\winboot directory over c:\wubildr seems to fix it. Or loop mounting the root.disk and manually editing the /boot/grub/grub.cfg file (just leave the parts you need to boot i.e. anything on or after the first  starting with "menuentry".
Note: There is no permanent known fix to this and the bug report has not been addressed by the developers since before the release. (Unlike for the 10.04 bugs, more proactive testing by community members identified the problems, early - but unfortunately once 10.10 was released the developers seem to have lost interest and moved on to Natty Narwhal 11.04).

So what can you do to avoid these problems?
Don't update grub-pc, grub-common or lupin-support packages. Don't play with grub unnecessarily. Don't upgrade to a new release without taking a full backup and having a repair CD handy.
The ultimate prevention is to migrate your Wubi install to a normal partition install.