Technical miscellanea by Derrick Coetzee

Windows 7 hanging with black screen on boot

This was a frustrating problem for me: Windows installed a mandatory security update and rebooted my system without asking. Then, the system won’t start back up. I get a logo screen, then I just get a black screen, with the CPU peaked (based on fan noise) and the disk light showing little to no activity. I try to boot into Safe Mode, and it tells me it’s rolling back updates. I try to boot into Safe Mode again, and it hangs at the “Please wait” screen and makes no progress. I anticipate that a System Repair will be needed, so I decrypt my volume and then do a System Repair, but Windows says there’s nothing to repair. I try to use System Recovery and I’m informed that there are no System Recovery checkpoints.

My next step is to try to run sfc to check system files, from the command prompt in the Windows 7 repair options. I run:

sfc /scannow /offwindir=d:\windows /offbootdir=d:\

I’m informed that:

“There is a system repair pending which requires reboot to complete. Restart Windows and run sfc again.”

This isn’t very helpful, since of course starting my system is the problem. However, I discovered that by renaming d:\windows\winsxs\pending.xml, I could eliminate the pending system repair and allow the sfc run to complete.

After doing this, probably just because I renamed pending.xml, I could successfully boot into Safe Mode. Now comes part 2. I went to Device Manager and, on a hunch, went to the most recent driver I’d updated, a video driver for my NVIDIA GeForce GT 230M, the Verde version 280.26 driver released on August 9, 2011. I had installed this driver a little while back and not rebooted until now, and a black screen often points to video driver issues, and I could boot in Safe Mode which doesn’t use the video driver, so it made sense that it might be behind it. I clicked properties and did “Roll Back Driver” to get back to version 275.33 which I was using before. After doing this, and rebooting, I could boot normally again. I went ahead and installed the latest Windows updates (not including the graphics update) and rebooted without incident.

I went ahead and left a post about this at the NVIDIA forums. Hopefully the bad driver will get fixed and a new minor version released soon.

Cleanly disconnecting an iSCSI disk in Windows 7

At home I have a NAS with 3 disks in a RAID 5 configuration that exposes an iSCSI volume that I use from Windows machines. I chose iSCSI so I could format it with NTFS and use advanced file system features that the NAS doesn’t support natively. Although NTFS does not support multiple clients connected to the same iSCSI volume at once, if I want multiple machines to use it simultaneously, I just connect it to one machine and share it out over Windows File Sharing / Samba / SMB.

However, although Windows 7 ships with iSCSI support, it makes it awkward to transfer it from one machine to another in the LAN, which is sometimes useful. If I’ve used the disk at all, it will inevitably decline to disconnect it with the following error message:

The session cannot be logged out since a device on that session is currently being used.

Even when I’ve closed all apps that could possibly be using the disk, the message persists. But it turns out there’s a simple way around this:

  1. Open up Control Panel -> Administrative Tools -> Computer Management, and in the left pane click on Storage -> Disk Management.
  2. Right-click the iSCSI disk (the disk itself, all the way to the left, not the logical partition) and choose “Offline”. This disconnects the disk. It is not the least bit deterred by applications actively using the disk (they will receive an error if they attempt to continue reading it).
  3. Proceed to disconnect the iSCSI volume normally. It will succeed without any delay.

When later re-connecting the disk back to the original machine, you must return to Computer Management, right-click the disk, and choose Online before it is accessible again. Remember when connecting to an iSCSI disk to not add it to your “Favorite Targets” because this tends to lead to multiple machines accidentally connecting to the same iSCSI disk at the same time later on.

The unconquerable “mount error 13 = Permission denied”

When you try to mount a Windows (SMB) share on a Linux machine using CIFS, the error reporting gives you very little insight into what could be causing the problem. This weekend, I encountered a mysterious and very persistent error “mount error 13 = Permission denied” whenever attempting to mount a share on my Windows 7 laptop. This error normally indicates that the credentials are invalid. This was strange because:

  • I was able to mount shares on another Windows 7 machine at home that was in the same workgroup and had a user account with the same username and password. (not a credentials issue)
  • The Windows 7 machine at home was able to access the shares on my laptop, and I was able to access other servers on my laptop from Linux. (not a firewall/networking issue).
  • Even a fresh VM running Ubuntu 10.10 32-bit experienced the same problem (not a Linux configuration issue, regression in Ubuntu 11.04, or 64-bit specific issue).
  • Whether the VM is run on the same machine or a different machine, whether it uses a private network or NAT or bridged connection, the same problem occurs.

I suspected a misconfiguration on my Windows 7 laptop – it was initially part of a domain, but removing it from that domain did not fix the problem. I checked the settings in the Local Group Policy Editor under Local Computer Policy -> Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> Security Options, and they matched those on the “good” machine. I also couldn’t find any information about the failed log-on attempts in the system log on either the server or client.

My solution, ultimately, was to reformat and reinstall Windows 7 on my laptop. This worked, suggesting that indeed something had got haywire with the Windows 7 configuration, although still I can’t fathom what it was. Hopefully if someone else runs into this problem they’ll come up with a better solution.

Mysterious VMware shutdowns when I lock my computer

I have an Ubuntu 11.04 VM running in VMware on my Windows machine, and twice this morning after locking my computer and going away for a few minutes, I came back to discover the VM had powered off. At first I suspected a system crash, but there were no errors in the system logs.

It turns out that the way I was locking my computer was to press CTRL+ALT+DEL, then press ENTER to choose “Lock this computer”. In VMware 7.1.4, the CTRL+ALT+DEL keypress is also sent to the VM when it has focus. In Ubuntu 11.04, this key is bound by default to the “Log out” action, which brings up the “Log out” window. If you leave the Log out window up for 60 seconds, it automatically shuts down the system.

The solution: go in Ubuntu to System->Preferences->Keyboard shortcuts, scroll down and click on Desktop->Log out, and press backspace to remove the key binding (or select another one of your choice). Another alternative is to use WINDOWS+L to lock your computer, which should be easier anyway.

Fixing missing VMware network connections

When you install VMware Workstation it creates several virtual network adapters for the purpose of managing its virtual network and giving VMs access to both private and public networks on the host. I recently encountered a problem with VMware Workstation 7.1.4 where some of these network adapters vanished, and consequently network access in the VM stopped working. I still have no idea how this occurred – it might be because I tend to disable VMware network connections when not using them to avoid interference in packet routing – but in any case no amount of tweaking my VM’s network settings seemed to fix it. I was unable to connect in the VM in NAT mode or Bridged Mode, so it couldn’t get Internet.

In the end I shut down the VM, opened up Edit -> Virtual Network Editor…, and clicked the “Restore Default” button, which not only restores the original Virtual Network settings but also uninstalls and reinstalls the network adapters. This fixed the problem with little trouble.

Setting up your own local Ubuntu mirror

The Ubuntu Linux package repository is a great resource, giving easy access to an enormous number of free applications, but on the other hand requires an Internet connection and quite a bit of bandwidth to use, especially for packages with many dependencies. This is problematic in any scenario where you’re offline (e.g. on a plane), on a low-bandwidth link, or on a link with high bandwidth cost. The best solution to this is to plan ahead: go to a place with low-cost bandwidth, like many cities in the US and Europe, and set up a local Ubuntu mirror on your local storage, as described below. If you can’t go there yourself, get a friend with a low-cost bandwidth link to put a mirror on an external storage device (a 2.5 inch external hard drive,a Bluray disc, a thumb flash drive, etc.) and drop it in the mail.

Using debmirror to produce the mirror:

  1. debmirror is designed for producing mirrors of Aptitude repositories like those of Debian and Ubuntu. Here’s the command I use, which has more than I need, strictly speaking (make sure to “mkdir ubuntu-packages” first):

    debmirror –progress –state-cache-days=365 –host=source.mirror.hostname.goes.here.org –method=rsync –root=ubuntu –dist=maverick,maverick-backports,maverick-proposed,maverick-security,maverick-updates –section=main,restricted,multiverse,universe –arch=i386,amd64 ubuntu-packages/ –ignore-release-gpg

    Here are the most important options to change:

    • Replace “source.mirror.hostname.goes.here.org” with a suitable mirror host from the Mirrors of Ubuntu list. I recommend you choose one with a high-speed link near your geographical location that has “rsync” next to it. It’s okay to choose a mirror which is not up-to-date – we’ll fix that in step 2.
    • Set “–method” to one of the methods supported by your server (http, ftp, or rsync). rsync is best if supported; http and ftp work but are much slower at transferring large numbers of small files.
    • Use “–arch=i386″ to include 32-bit binary packages, “–arch=i386″ to include 64-bit binary packages, and “–arch=i386,amd64″ for both
    • Add “–nosource” if you don’t need source packages (you don’t, unless you’re planning to recompile packages from scratch).
    • In “–dist” feel free to remove maverick-backports and maverick-proposed if you’re only interested in mainline Ubuntu packages and not more cutting-edge recent packages.


    The initial download may take a few days, depending on your bandwidth. You can interrupt the download at any time with CTRL+C and resume it later by re-running the command.

    You may receive “key could not be found” errors. You can safely ignore them. If you want to get rid of them anyway, see the FILES section at the end of the debmirror man page.

  2. After you eventually finish, you can modify the mirror host and set it to a different host labelled “Up to date” on the Mirrors of Ubuntu list and run the command again. This will ensure you have the most recent versions of all packages.
  3. You can re-run the debmirror command later to update an existing mirror with new updates. Be warned that once you’ve downloaded a mirror, you have to return yourself or your external storage to the high bandwidth environment to update it (or else you have to download them normally) – attempting to download updates over an expensive connection will quickly prove more costly than downloading packages you need directly. This can be a downside, but having stale versions for a little while sure beats having no versions.
  4. Remember, once you’ve downloaded a local repo, the bandwidth cost of giving a copy to a friend is zero! Ask anybody who might want one.

At the present time, all packages including i386 binaries, x86-64 binaries, and sources comes to about 89 gigabytes. Just including binaries for your platform will probably be about 30 GB, small enough to fit on a Bluray or 64 GB flash drive (it might fit on 32 GB but that might be cutting it a bit too close).

Configuring your installation to use the mirror (the following steps require root access):

  1. Mount or copy the package mirror at some location like “/usr/packages”.
  2. Make a backup of your /etc/apt/sources.list file as /etc/apt/sources.list.remote. Modify your /etc/apt/sources.list to replace every HTTP or FTP URL with “file:///usr/packages” (or whatever your local path is).
  3. Do an “apt-get update”, then go ahead and use all your package manager software normally to upgrade or install packages. It will appear to skip the download step and proceed directly to installation.

Another reason you might want a local mirror is if you administer a number of Ubuntu installations, each with different package requirements. This might be several VMs on your own machine, you and your roommates’ machines at home, sharing over wifi with your neighbors, or all the machines in a small office. However, be warned that if your repository is not accessed frequently enough, the bandwidth it costs to update it will never justify the bandwidth you save by using it.

Getting Bugzilla e-mail working on Amazon EC2

If you naively try to install Bugzilla (or any other app that sends e-mail)  on your Amazon EC2 server, you’ll soon discover that EC2′s entire IP block has been blacklisted by spam filters worldwide, probably due to abuse by a small number of customers. To get around this, you’ll have to take advantage of Amazon’s Simple Email Service (SES), which was designed to not end up getting blacklisted in the same manner by giving Amazon careful control over the content that gets sent out and the rate at which it’s sent. Among other things, it validates from addresses and performs egress spam filtering on the way out. However, setting up SES is not as simple as pointing Bugzilla at some Amazon-internal SMTP server – instead you have to follow the following steps (the following steps were done for Bugzilla 3.6.2.0-3):

  1. Go to http://aws.amazon.com/ses/ and click “Sign up” on the right-hand side (EC2 customers are not automatically signed up for SES).
  2. Follow the directions at Email Address Verification in the Amazon SES Developer Guide to verify the e-mail address that Bugzilla will be sending mail from. This address must be able to receive mail so that you can click the link in the confirmation e-mail. You’ll need this e-mail in step 7 below.
  3. Verify another e-mail address in the same manner that you will be using to create a test account on Bugzilla with. An account should not already exist for this e-mail address. (This is needed because until you request production access in step 9, you can only send e-mail to verified addresses.)
  4. Take the Perl scripts you downloaded in step 2 along with the aws-credentials file you created and install them at “/opt/third-party/amazon”. Add a user called “aws-email” with adduser and chown these files to that user. Move the Perl module SES.pm to a location on your default Perl include path – on my system I used /usr/lib/perl5/.
  5. Modify your local Postfix or Sendmail configuration according to the directions at Integrating with Your Existing Email Server in the Amazon SES Developer Guide. This will cause all mail sent from your server to go through SES. For the username “user” use aws-email.
  6. SSH into your server and find the Bugzilla Perl module Mailer.pm. For me this was under /usr/share/perl5/Bugzilla. Edit it and comment out the line reading “$email->header_set(‘Auto-Submitted’, ‘auto-generated’);” (line 94 in my version). This header is not currently supported by the SES tools.
  7. Log into Bugzilla with an administrative user and click Administration, Parameters, Email. For mail_delivery_method choose “Sendmail” (this will work even if you’re using Postfix). For mailfrom, put the address you used in step 2. Click Save Changes.
  8. Log out of Bugzilla, make sure your Postfix server is started, and create a new user account using the test e-mail address from step 3. This e-mail address should receive the Bugzilla confirmation e-mail successfully and it should not be filtered as spam. If it does not, check your mail log (usually at /var/log/mail.log) to see what went wrong. If you need to test it again, you may have to verify another, different e-mail address and use that one, because Bugzilla enforces a delay before it lets you try to use the same e-mail to sign up again. I found it useful to use addresses of the form “username+extension@domain.tld”, which appear different each time you change the extension but are still sent to “username@domain.tld”.
  9. Finally, Request Production Access for SES. This will take between a few hours and a few days and will enable Bugzilla to send to e-mail addresses that have not been verified as in steps 2 and 3.

If you encounter any trouble with these steps, please post here and the other commenters and I can help figure out what’s going on.