Technical miscellanea by Derrick Coetzee

Redirecting access log with .htaccess

Written by dcoetzee on 2012-02-11 — Leave a Comment

I had an account on a seedbox server with a shared web server. The access logs were off limits to me, and the admins weren’t about to reconfigure the server for me. But I really wanted to see who was accessing my site. What to do?

Turns out I do have .htaccess on this server. However, the access log is not one of the configurable parameters. But through a hackish combination of several tricks, I can log all accesses to my public_html tree. Here’s my .htaccess:

RewriteEngine on
RewriteCond %{REQUEST_URI} !^/hitcounter.php
RewriteRule ^ http%1://www.%{HTTP_HOST}/hitcounter.php?uri=%{REQUEST_URI} [L,R=301]

It redirects all incoming accesses to the “hitcounter.php” script, which logs the access to a file called “log” in the usual Apache access log format:

<?php
if (getenv(HTTP_X_FORWARDED_FOR)) {
    $ip = getenv(HTTP_X_FORWARDED_FOR);
    $vip = getenv(REMOTE_ADDR);
} else {
    $ip = getenv(REMOTE_ADDR);
}
$uri = $_GET['uri'];
$date = date('d/m/Y:H:i:s O');
$fh = fopen('log', 'a');
fwrite($fh, "$ip - - [$date] \"GET $uri HTTP/1.0\" 200 -\n");
fclose($fh);

header("Location: /nocount$uri");
?>

At the end of the script it redirects to the original URL prepended with “/nocount”. The directory “/nocount” simply contains symbolic links to everything in the webroot, along with a new .htaccess that turns rewriting back off again:

RewriteEngine off

One handy feature of this is I can bookmark or give out links containing “nocount” and those won’t be logged, so I can avoid self-logging. Downside is, this particular scheme results in “/nocount” being prepended to all my URLs.

I ended up abandoning this scheme because I was using my web server to web seed torrents and these redirects confused and broke the uTorrent client somehow. It also turned out to be unnecessary because Feral Hosting actually lets me configure my own private Apache server even on shared hosts via the .apache2 directory in my home directory.

Posted in Uncategorized

Installing Eclipse with PyDev for Python development in Ubuntu

Written by dcoetzee on 2011-11-23 — Leave a Comment

The following instructions been tested with Ubuntu 11.04 (Natty Narwhal), Eclipse 3.5.2, Python 2.7.1+, and PyDev 2.2.1, all of which were current on 2011 August 22. They were later tested again with Ubuntu 11.10 (Oneiric Ocelot).

Installation part based on this blog entry, with modifications.

Installing Sun JDK

First, assuming you want to use the Sun JDK, we need to get access to the Sun JDK package. Uncomment this in your /etc/apt/sources.list:

deb http://archive.canonical.com/ubuntu natty partner
deb-src http://archive.canonical.com/ubuntu natty partner

And do this (from this post):

sudo add-apt-repository ppa:sun-java-community-team/sun-java6
sudo apt-get update

Install Eclipse and Sun JDK and plug-in:

sudo apt-get install eclipse sun-java6-jdk sun-java6-jre sun-java6-bin sun-java6-plugin

Accept license terms when prompted. After this completes, make Sun’s JDK the default:

sudo update-java-alternatives -s java-6-sun

Installing PyDev

  1. Run Eclipse and then go to Help | Install New Software
  2. Click Add… next to the “Work with” drop down. Specify the following:
    Name: PyDev
    URL: http://pydev.org/updates
  3. Check “PyDev”. Do not check “Pydev Mylyn Integration”. Press Next.
  4. Press Next again, accept license terms, click Finish to begin install. Installation may proceed in background – see lower-right of the window for status.
  5. If prompted to accept a certificate, click Select All, then OK. If prompted to restart Eclipse, click Yes.
  6. Open Windows | Preferences | Pydev | Interpreter – Python.
  7. Click Auto Config. A “selection needed” window will appear for the SYSTEM pythonpath. Click Select All, click OK.
  8. When Auto Config completes, click OK and wait for Pydev to scan your system Python libraries.

PyDev installation is now complete.

Trying out PyDev

To create a new (fresh) Python project in Eclipse with PyDev:

  1. Click File | New | Project | PyDev | PyDev Project, and click Next.
  2. Enter a project name. Click Finish.
  3. An Open Associated Perspective dialog will appear. Check the checkbox and click Yes.
  4. Right-click “src” and choose New | PyDev Module. Enter a Name (you need not enter a Package). Click Finish.
  5. Type some Python into the text file, like:
    print(‘hello world’.capitalize())
    You can press ENTER to autocomplete the capitalize() call while typing it.
  6. Click Run | Run. Leave set to “Python Run” and click OK. A Console window tab should appear showing the program output.
  7. Add the following two lines of source code:
    x = ‘hello there’
    print(x)
    Right-click in the margin to the left of the first source code line and choose “Add Breakpoint”.
  8. Choose Run | Debug. When “Confirm Perspective Switch” appears, check the box and click Yes.
  9. Execution will break at the first line. Select Run | Step Over, click the “Step Over” toolbar button, or press F6 to step through the source code line by line. After stepping over the line “x = ‘hello there’” the variable x should appear in the “Variables” window (you may have to scroll it down). Use Run | Debug or F11 to start the program over again during debugging.
  10. When done debugging, click the “PyDev perspective” in the upper right hand corner to return to the normal source editing view.
  11. To use PyDev with an existing Python project you’re working on, see “Creating a Project from existing sources“.

Keyboard shortcuts

Important keyboard shortcuts to know with PyDev:
  • F3: Go to definition
  • CTRL+SHIFT+G: Find all references (this one isn’t even in the menus!)
  • CTRL+SHIFT+B: Add breakpoint
  • F11: Run in debug mode
  • F6: Step over
  • F5: Step into
See Appcelerator PyDev keyboard shortcut list for more, or press CTRL+SHIFT+L.
Posted in Uncategorized

Installing Ruby 1.9.2 and gems on Ubuntu 11.10

Written by dcoetzee on 2011-10-25 — Leave a Comment

By default, the “ruby” package on Ubuntu 11.10 (Oneiric) refers to an older version of Ruby, 1.8.7 – probably for compatibility reasons. To use the newest version of Ruby and Gem, do:

  • sudo apt-get install ruby1.9.1
  • update-alternatives --config ruby
  • (select the option reading “/usr/bin/ruby1.9.1″)
  • update-alternatives --config gem
  • (select the option reading “/usr/bin/gem1.9.1″)

Contrary to their names, /usr/bin/ruby1.9.1 is actually a newer version (1.9.2p290 at the present time) and /usr/bin/gem1.9.1 is actually RubyGems 1.3.7 (there is no RubyGems 1.9.1). These numbers refer to the package name ruby1.9.1.

During a “bundle install”, depending on what gems are being installed, you may encounter other errors. I encountered the following three:

  • ERROR: Error installing nokogiri:
ERROR: Failed to build gem native extension.

/usr/bin/ruby1.9.1 extconf.rb
<internal:lib/rubygems/custom_require>:29:in `require': no such file or load -- mkmf (LoadError)
from <internal:lib/rubygems/custom_require>:29:in `require`
from extconf.rb:5:in `<main>'

    To fix this one do: apt-get install ruby1.9.1-dev

    ERROR: Error installing nokogiri:
ERROR: Failed to build gem native extension.

/usr/bin/ruby1.9.1 extconf.rb
checking for libxml/parser.h... no
-----
libxml2 is missing.  please visit http://nokogiri.org/tutorials/installing_nokogiri.html for help with installing dependencies.
-----

    To fix this one do: apt-get install libxml2-dev

    ERROR: Error installing nokogiri:
ERROR: Failed to build gem native extension.

/usr/bin/ruby1.9.1 extconf.rb
checking for libxml/parser.h... yes
checking for libxslt/xslt.h... no
-----
libxslt is missing.  please visit http://nokogiri.org/tutorials/installing_nokogiri.html for help with installing dependencies.
-----

    To fix this one do: apt-get install libxslt1-dev

  • Installing sqlite3 (1.3.4) with native extensions
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.
/usr/bin/ruby1.8 extconf.rb
checking for sqlite3.h... no
sqlite3.h is missing. Try 'port install sqlite3 + universal'
or 'yum install sqlite3-devel' and check your shared library search path (the
location where your sqlite3 shared library is located).

    To fix this one do: sudo apt-get install libsqlite3-dev

  • Gem::InstallError: hoe requires RubyGems version >= 1.4. Try 'gem update --system' to update RubyGems itself.

    Version 1.9.2 of the “hoe” gem requires a newer version of RubyGems than is presently available via the Ubuntu package manager. You have to force an upgrade with:

    sudo REALLY_GEM_UPDATE_SYSTEM=1 gem update --system

    After doing so, you must redo “gem install bundler” and “bundle install”. Note that this might screw up other things, since you’re upgrading gem outside the package manager interface now.

If these instructions don’t work for you, or you encounter problems with other gems and figure out how to fix them, please post in the comments below!

Posted in Uncategorized

Getting the ASUS USB-N13 working under FreeNAS

Written by dcoetzee on 2011-10-21 — 2 Comments

I set up a new FreeNAS system today that is dual-homed, with a wired connection to an isolated Gigabit LAN (for performance), and a wireless connection to the Internet (for updates, management, etc.). Since my HP ProLiant MicroServer did not come with built-in wireless, I got a USB wireless card, the ASUS-N13, that has a good record of compatibility with Linux and FreeBSD.

However out-of-the-box, FreeNAS is a very minimal distribution, missing a lot of kernel modules needed to make this device work. To fix this, I followed these steps:

  • Download the installer ISO for the matching version and platform of FreeBSD.
  • Mount the ISO and dug into /boot/kernel
  • Use “mount -uw /” on the NAS to remount the root as read/write so you can write to /boot and /conf.
  • Use scp to upload the following kernel modules from the FreeBSD ISO to your running FreeNAS install under /boot/kernel:
    • if_run.ko runfw.ko wlan*.ko
  • Edit /boot/loader.conf and add the lines: 
    if_run_load="YES"
runfw_load="YES"
  • Go to /conf/base/etc and edit/create the necessary files as described in the FreeBSD Handbook Wireless Networking Guide. I have a WPA-authenticated network so I created wpa_supplicant.conf and edited rc.conf to add: 
    wlans_run0="wlan0"
ifconfig_wlan0="WPA DHCP"

    Note the use of “run0” here, since this device uses the “run” kernel module.

  • Copy your modified files from /conf/base/etc to /etc and test with: 
    /etc/rc.d/netif start

    If you connect successfully, ping a host like google.com to confirm connectivity.

  • Reboot. Log back in. Use /sbin/ifconfig and ping to confirm you are still connected.
Posted in Uncategorized

Running Ubuntu Server 11.10 Linux on the Thecus N7700 NAS

Written by dcoetzee on 2011-10-17 — 1 Comment

A couple years ago I bought the Thecus N7700, a low-end NAS with 7 hot swappable SATA disks. I wanted an affordable solution that would give me a lot of capacity at home for raw photos, videos, and ISOs/VMs, so I filled it with consumer-grade 2 TB disks in RAID 5 configuration. Unfortunately, I would soon discover that Thecus has a very short “compatibility list” of drives they test with, all of them enterprise-grade disks, and the drives I bought were not on it. On at least a dozen occasions, the system reported I/O errors with one or more disks. I would reboot and rebuild the RAID, then check the SMART data, just to discover the disks had experienced no errors at all – it was a firmware issue. Despite releasing many updates to the firmware since then (several of which claimed to fix this issue), the problem persists.

Fortunately, the machine is x86 and made entirely of commodity parts, so it’s possible to install any OS I like on it. The catch is, it has no VGA out. However, it does have integrated video on the motherboard – they just didn’t solder a connector onto it. Following the directions here, I disassembled one end of a VGA cable (first I used a flathead screwdriver to push off the plastic cover, then used some needlenose pliers to twist the metal tabs until they broke off, exposing the pins). I then poked the pins through a small piece of paper, for insulation, and inserted them directly into the motherboard. The VGA holes are at the back of the motherboard near the other connectors. I was successful in getting VGA out, although it took quite a bit of fiddling with the angle of the cable to make it touch things just right – in the future I may want to solder a connector onto the board for convenience. Once I’d done this, I could plug in a USB keyboard and hit DEL to enter BIOS at boot and turn on USB booting – note that you do not want to set USB-CDROM/USB-FDD etc. as the boot device, but rather you must plug in a USB flash drive first, which the BIOS will correctly see as a disk device, and then change the priority of boot disk devices. After that, I could set up the USB flash drive with any installer or OS I wanted – I used an 8 GB flash drive. There are alternative ways to getting another OS to run on the board – like replacing the internal 128 MB flash drive – but it’s much easier to diagnose boot issues if you can see the VGA out.

I started out by installing FreeNAS, just to discover it doesn’t support the N7700′s SATA controller. Next I tried Openfiler but couldn’t get it to boot from USB (although this might be possible). Finally I decided to go with the freshly-released Ubuntu Server 11.10 Linux. I used Unetbootin to put its installer ISO onto my USB key, and sure enough it could see all my disks. Catch is, I wanted to install it to the USB key, but I was using that for the installer. The internal flash drive was too small to install to (128 MB). The Thecus N7700 only has two USB ports – if I had a second USB flash drive and a USB hub handy, I would’ve just installed directly onto the second USB key (I recommend trying this), but I didn’t, so instead I installed onto a small root partition on one of my RAID hard drives, sized to match the USB flash drive. It failed to install Grub to the MBR during install, but I ignored this. Afterwards, I attached both the flash drive and the disk to my PC, repartitioned the flash drive in Parted Magic, and then used Clonezilla to copy the installed partition from my hard disk to the flash drive. There was a small catch to this: I had to create a tiny 1 MB partition before my Ubuntu partition and mark it as a BIOS boot partition for Grub. Then, I used the Ubuntu 11.10 Desktop Live CD in live mode to chroot into the Ubuntu filesystem (see this guide to chrooting involving mount -o bind) and run “install-grub /dev/sdb”, where /dev/sdb was my USB flash device, and “update-grub”. At this point the device was bootable.

Once I had booted up the device, I had to fix some annoyances. First of all, device names of disks can change between boots, particularly if you remove or add disks. To deal with this I created /etc/udev/rules.d/nas-disk.rules with the following contents:

SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", ENV{ID_PATH}=="pci-0000:04:00.0-sas-0x0000000000000000-lun-0", SYMLINK+="nasa"
SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", ENV{ID_PATH}=="pci-0000:04:00.0-sas-0x0100000000000000-lun-0", SYMLINK+="nasb"
SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", ENV{ID_PATH}=="pci-0000:04:00.0-sas-0x0200000000000000-lun-0", SYMLINK+="nasc"
SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", ENV{ID_PATH}=="pci-0000:04:00.0-sas-0x0300000000000000-lun-0", SYMLINK+="nasd"
SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", ENV{ID_PATH}=="pci-0000:05:00.0-sas-0x0000000000000000-lun-0", SYMLINK+="nase"
SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", ENV{ID_PATH}=="pci-0000:05:00.0-sas-0x0100000000000000-lun-0", SYMLINK+="nasf"
SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", ENV{ID_PATH}=="pci-0000:05:00.0-sas-0x0200000000000000-lun-0", SYMLINK+="nasg"

Now, /dev/nasa thorugh /dev/nasg referred to my seven disks in order from the top down. Warning: if you mount any volumes in your /etc/fstab, make sure you include the “nobootwait” option. Otherwise, if a volume fails to mount for any reason during boot, it will just sit there waiting for you to press S, before even starting the SSH server.

Once I had done this I installed a number of useful packages:

  • sshd: Already installed during installation, lets me administer the machine over the network with the box closed and no VGA cable attached.
  • mdadm: Needed for software RAID support.
  • xfsprogs reiserfsprogs btrfs-tools hfsprogs: Tool support for additional filesystems.
  • nfs-kernel-server: Lets me share files with other UNIX machines over NFS. (see this guide – I found it useful to set no_root_squash so my client had free access)
  • samba: Installed during installation, lets me share files with Windows clients.
  • webmin: Gives me a remote web administration interface on port 10000 that I can use to, among other things, create, modify, and destroy RAID partitions.
  • smartmontools: Allows me to regularly check the SMART data of my disks and run self-tests, and report any problems.
  • transmission-daemon: Provides a web interface for BitTorrent downloads. Requires some configuration, see this guide.
  • sysstat: Includes iostat, which lets me view current disk activity.
  • open-iscsi open-iscsi-tools: iSCSI support (not currently using this but I used to use iSCSI exclusively and manage the volume as NTFS on Windows clients).

At first I considered running btrfs on my NAS, which is a very promising up-and-coming competitor to ZFS, but there’s no official stable release quite yet and I’m uncertain about performance. ZFS itself is only available in alpha form on Linux, primarily due to licensing issues. I ended up running XFS because it’s mature and handles volumes with many large files well.

Some notes on RAID: I used software RAID 5 over 5 disks with mdadm. It starts in degraded state, then rebuilds, and will send mail to root about this, so don’t be alarmed. You can use it right away if you want (although this slows down rebuild), and you can monitor the rebuild process via SSH with while true; do cat /proc/mdstat; sleep 5; done.

Note that if you do modify the OS of your Thecus N7700, the disk LEDs and LCD display will no longer work. The LCD display will show “Running self tests” forever, and the blue and red LEDs will also remain on and not flashing indefinitely. I believe a sufficiently clever person could reverse engineer their interface and provide a command-line utility or daemon to control them, which would be quite handy, but I don’t need it. The power and network LEDs should still work fine.

Note that if you insert or remove drives, or boot without the USB flash drive attached, you may accidentally alter the disk boot priority order in BIOS, causing the internal flash drive to gain priority, or causing the flash drive to fall off the list. If this happens, you still need the VGA cable to get back into BIOS to fix it. To mitigate this problem, I recommend completely erasing the internal flash drive by filling it with zeros: dd if=/dev/zero of=/dev/sda bs=128M (make sure it’s /dev/sda on your machine though!)

Warning: When you close back up the case, remember to plug back in the case fans. Seven disks generate a lot of heat and you will fry your machine if you don’t do this. Sadly I destroyed my Thecus this way only a single day after getting my RAID set up, so I can’t say much about long-term use.

Let me know if you have any problems with this, and please share your own experiences! To close this out, here is a bunch of data about the Thecus N7700 system that you may find useful if you’re trying to get another distro or OS running on it:

lspci:

00:00.0 Host bridge: Intel Corporation Mobile 945GME Express Memory Controller Hub (rev 03)
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GME Express Integrated Graphics Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 1 (rev 02)
00:1c.2 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 3 (rev 02)
00:1c.3 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 4 (rev 02)
00:1c.4 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 5 (rev 02)
00:1c.5 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 6 (rev 02)
00:1d.0 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #3 (rev 02)
00:1d.3 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #4 (rev 02)
00:1d.7 USB Controller: Intel Corporation N10/ICH 7 Family USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e2)
00:1f.0 ISA bridge: Intel Corporation 82801GHM (ICH7-M DH) LPC Interface Bridge (rev 02)
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation N10/ICH 7 Family SMBus Controller (rev 02)
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
04:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE6440 SAS/SATA PCIe controller (rev 02)
05:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE6440 SAS/SATA PCIe controller (rev 02)

/proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 14
model name      : Intel(R) Celeron(R) M CPU        440  @ 1.86GHz
stepping        : 12
cpu MHz         : 1866.707
cache size      : 1024 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov clflush dts acpi mmx fxsr sse sse2 ss tm pbe nx constant_tsc up arch_perfmon bts aperfmperf pni monitor tm2 xtpr pdcm dts
bogomips        : 3733.41
clflush size    : 64
cache_alignment : 64
address sizes   : 32 bits physical, 32 bits virtual
power management:

/proc/meminfo

MemTotal:        1023884 kB

lsmod

Module                  Size  Used by
serio_raw              12990  0
i915                  505108  1
drm_kms_helper         32889  1 i915
drm                   192226  2 i915,drm_kms_helper
i2c_algo_bit           13199  1 i915
video                  18908  1 i915
lp                     17455  0
parport                40930  1 lp
raid10                 30270  0
usb_storage            44173  1
raid456                61518  1
async_raid6_recov      12906  1 raid456
async_pq               12959  2 raid456,async_raid6_recov
uas                    17699  0
ahci                   21634  0
libahci                25727  1 ahci
mvsas                  51615  8
libsas                 62786  1 mvsas
scsi_transport_sas     34482  2 mvsas,libsas
e1000e                139775  0
raid6_pq               88205  2 async_raid6_recov,async_pq
async_xor              12738  3 raid456,async_raid6_recov,async_pq
xor                    21860  1 async_xor
async_memcpy           12481  2 raid456,async_raid6_recov
async_tx               13123  5 raid456,async_raid6_recov,async_pq,async_xor,async_memcpy
raid1                  26291  0
raid0                  17067  0
multipath              12977  0
linear                 12792  0

smartctl -a /dev/hda (info about the internal flash drive)

Device Model:     128MB ATA Flash Disk
Serial Number:    C191100004A12b39GGwP
Firmware Version: AD B512D
User Capacity:    128,057,344 bytes [128 MB]
Sector Size:      512 bytes logical/physical
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 3a
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/proc/ioports

0000-0cf7 : PCI Bus 0000:00
  0000-001f : dma1
  0020-0021 : pic1
  0040-0043 : timer0
  0050-0053 : timer1
  0060-0060 : keyboard
  0064-0064 : keyboard
  0070-0073 : rtc0
  0080-008f : dma page reg
  00a0-00a1 : pic2
  00c0-00df : dma2
  00f0-00ff : fpu
  0170-0177 : 0000:00:1f.1
    0170-0177 : ata_piix
  01f0-01f7 : 0000:00:1f.1
    01f0-01f7 : ata_piix
  02f8-02ff : serial
  0376-0376 : 0000:00:1f.1
    0376-0376 : ata_piix
  03f6-03f6 : 0000:00:1f.1
    03f6-03f6 : ata_piix
  03f8-03ff : serial
  0400-04bf : pnp 00:08
    0400-0403 : ACPI PM1a_EVT_BLK
    0404-0405 : ACPI PM1a_CNT_BLK
    0408-040b : ACPI PM_TMR
    0428-042f : ACPI GPE0_BLK
  04d0-04d1 : pnp 00:01
  0500-051f : 0000:00:1f.3
  0680-06ff : pnp 00:01
  0880-088f : pnp 00:01
0cf8-0cff : PCI conf1
0d00-ffff : PCI Bus 0000:00
  8000-8fff : PCI Bus 0000:04
    8f00-8f7f : 0000:04:00.0
      8f00-8f7f : mvsas
  9000-9fff : PCI Bus 0000:03
    9f00-9f1f : 0000:03:00.0
  a000-afff : PCI Bus 0000:02
    af00-af1f : 0000:02:00.0
  b000-bfff : PCI Bus 0000:01
  c000-cfff : PCI Bus 0000:05
    cf00-cf7f : 0000:05:00.0
      cf00-cf7f : mvsas
  d000-dfff : PCI Bus 0000:06
  f300-f30f : 0000:00:1f.2
    f300-f30f : ahci
  f400-f403 : 0000:00:1f.2
    f400-f403 : ahci
  f500-f507 : 0000:00:1f.2
    f500-f507 : ahci
  f600-f603 : 0000:00:1f.2
    f600-f603 : ahci
  f700-f707 : 0000:00:1f.2
    f700-f707 : ahci
  f800-f80f : 0000:00:1f.1
    f800-f80f : ata_piix
  fb00-fb1f : 0000:00:1d.3
    fb00-fb1f : uhci_hcd
  fc00-fc1f : 0000:00:1d.2
    fc00-fc1f : uhci_hcd
  fd00-fd1f : 0000:00:1d.1
    fd00-fd1f : uhci_hcd
  fe00-fe1f : 0000:00:1d.0
    fe00-fe1f : uhci_hcd
  ff00-ff07 : 0000:00:02.0

/proc/iomem

00000000-0000ffff : reserved
00010000-0009a7ff : System RAM
0009a800-0009ffff : reserved
000a0000-000bffff : PCI Bus 0000:00
  000a0000-000bffff : Video RAM area
000c0000-000dffff : PCI Bus 0000:00
  000c0000-000c7fff : Video ROM
000e0000-000effff : pnp 00:0b
000f0000-000fffff : reserved
  000f0000-000fffff : System ROM
00100000-3fddffff : System RAM
  01000000-01534783 : Kernel code
  01534784-017bbd7f : Kernel data
  01870000-01946fff : Kernel bss
3fde0000-3fde2fff : ACPI Non-volatile Storage
3fde3000-3fdeffff : ACPI Tables
3fdf0000-3fdfffff : reserved
3fe00000-febfffff : PCI Bus 0000:00
  d0000000-dfffffff : 0000:00:02.0
  e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
    e0000000-efffffff : reserved
      e0000000-efffffff : pnp 00:0a
  fd300000-fd3fffff : PCI Bus 0000:02
  fd400000-fd4fffff : PCI Bus 0000:02
    fd4c0000-fd4dffff : 0000:02:00.0
      fd4c0000-fd4dffff : e1000e
    fd4fc000-fd4fffff : 0000:02:00.0
      fd4fc000-fd4fffff : e1000e
  fd500000-fd5fffff : PCI Bus 0000:01
  fd600000-fd6fffff : PCI Bus 0000:06
  fd700000-fd7fffff : PCI Bus 0000:06
  fd800000-fd8fffff : PCI Bus 0000:01
  fd900000-fd9fffff : PCI Bus 0000:05
    fd900000-fd93ffff : 0000:05:00.0
  fda00000-fdafffff : PCI Bus 0000:05
    fdaff000-fdafffff : 0000:05:00.0
      fdaff000-fdafffff : mvsas
  fdb00000-fdbfffff : PCI Bus 0000:04
    fdb00000-fdb3ffff : 0000:04:00.0
  fdc00000-fdcfffff : PCI Bus 0000:04
    fdcff000-fdcfffff : 0000:04:00.0
      fdcff000-fdcfffff : mvsas
  fdd00000-fddfffff : PCI Bus 0000:03
  fde00000-fdefffff : PCI Bus 0000:03
    fdec0000-fdedffff : 0000:03:00.0
      fdec0000-fdedffff : e1000e
    fdefc000-fdefffff : 0000:03:00.0
      fdefc000-fdefffff : e1000e
  fdf00000-fdf7ffff : 0000:00:02.0
  fdf80000-fdfbffff : 0000:00:02.0
  fdffc000-fdffc3ff : 0000:00:1f.2
    fdffc000-fdffc3ff : ahci
  fdfff000-fdfff3ff : 0000:00:1d.7
    fdfff000-fdfff3ff : ehci_hcd
fec00000-ffffffff : reserved
  fec00000-fec003ff : IOAPIC 0
  fed13000-fed1dfff : pnp 00:0b
  fed20000-fed8ffff : pnp 00:0b
  fee00000-fee00fff : Local APIC
    fee00000-fee00fff : pnp 00:0b
  ffb00000-ffb7ffff : pnp 00:0b
  fff00000-ffffffff : pnp 00:0b

/proc/interrupts

           CPU0
  0:        304   IO-APIC-edge      timer
  1:          4   IO-APIC-edge      i8042
  8:          1   IO-APIC-edge      rtc0
  9:          0   IO-APIC-fasteoi   acpi
 14:      23215   IO-APIC-edge      ata_piix
 15:          0   IO-APIC-edge      ata_piix
 16:   88518197   IO-APIC-fasteoi   uhci_hcd:usb5, mvsas, i915
 17:   92740547   IO-APIC-fasteoi   mvsas
 18:          0   IO-APIC-fasteoi   uhci_hcd:usb4
 19:          0   IO-APIC-fasteoi   uhci_hcd:usb3
 23:    1014224   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
 48:          0   PCI-MSI-edge      ahci
 49:     745524   PCI-MSI-edge      eth1-rx-0
 50:     667535   PCI-MSI-edge      eth1-tx-0
 51:          3   PCI-MSI-edge      eth1
NMI:          0   Non-maskable interrupts
LOC:   19023450   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:          0   IRQ work interrupts
RES:          0   Rescheduling interrupts
CAL:          0   Function call interrupts
TLB:          0   TLB shootdowns
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
MCE:          0   Machine check exceptions
MCP:        262   Machine check polls
ERR:          0
MIS:          0
Posted in Uncategorized

What to do when there’s no MD5

Written by dcoetzee on 2011-10-09 — Leave a Comment

I recent downloaded a large number of very large files (mostly Linux distribution ISOs), and wanted to verify that the downloads were successful. I also wanted to have a hash file on hand to re-verify the downloads later on, in case of disk corruption. Many distributions include MD5, SHA1, or SHA256 hash files with their distributions right on the download web page, but many others don’t. If you encounter a situation like this, there are several ways to work around the problem:

  1. Right-click on the download link for the large file and then strip components of the URL to view the directory it’s in. Often that same directory will contain either a hash file for that large file (e.g. blah.iso.md5 or blah.md5), or a hash file for all the files in the directory (MD5SUMS). Similarly, try taking the filename and adding “.md5″ or replacing the extension with “.md5″ (or the same with .sha1) and then Googling that to see if it’s hiding somewhere else.
  2. If the file is hosted on SourceForge, SourceForge provides SHA1 and MD5 hashes for every file on their site. Just go to the “Files” section of the project, find the directory the file is in, and then click the “i” information bubble to the right of the filename.
  3. If there is a Torrent for the file, either download using that in the first place, or just open the Torrent file and point it at your existing file for seeding. Not only will this verify your file, but any corruption will be detected and fixed at the block level, so you don’t have to re-download the whole file. Once you’ve verified the file with your Torrent client, you can safely use a hash tool like md5sum to compute your own hash file for it.
  4. If the file is a compressed archive, such as a ZIP, .gz, .bz2, .xz, or .7z file, attempt to decompress it (when using gunzip/bunzip2/xz, make sure to pass –keep to keep the original input file around, because you might still need its hash later). A ISO may also contain hash files for files inside the image – check for these by using “mount -o loop” (or Daemon Tools/Virtual CloneDrive under Windows). There’s not an fsck for ISO 9660 that I know of, but you can scan through an ISO filesystem by using a command like “grep -r randomstring *” or “findstr /s randomstring *” in the root of the mounted ISO, where “randomstring” is a string that does not occur in any file. If any of these tests fail, the image is corrupt (but if they succeed, it does not necessarily prove that it is valid).
  5. If none of the above apply, things get tricky. Compute the MD5 and SHA1 hashes of the file using md5sum and sha1sum, then type them into Google. If you get any search results, it is extremely likely that your file is valid and someone else who downloaded it has posted their hash on a forum or digest listing (any false positives would indicate a hash collision, which these hashes are designed to make very unlikely).
  6. Finally, suppose you get no Google results. At this point, your only recourse is to re-download the file and compare the SHA1 of the two files. If they are identical, your file is good (provided of course the server’s copy is good – use an official server for at least one of the downloads). If they are different, redownload it again. Continue to redownload it until you get at least two files of the correct size with identical hashes. At this point you can be reasonably assured the file is valid (since two invalid versions are usually different from one another). If you get this far, be sure to post your MD5, SHA1, and SHA256 hashes, along with the original URL and a description of the file, on a public forum so that others can benefit. You might even create and publish a torrent for the file on a relevant torrent tracking website.
At first I was quite surprised that downloads can have invalid hashes at all, since TCP is supposed to ensure reliable transport. Unfortunately, HTTP/FTP downloads have a bad habit of either truncating early due to timeout, so it really is important to verify – at least 10% of my non-Torrent downloads were bad. However, most of them were merely truncated and could be resumed safely using “wget -c”, so at least I didn’t have to start the downloads over.
What’s a good way to fix the problem of files having no hashes available? Re-downloading files repeatedly is no fun if you’re paying per megabyte. These problems could be solved in principle using a special custom download manager. Here’s my basic design:
  • Whenever someone downloads a file, they would upload the URL and a hash tree for the file to a central server. If that hash is already on the server for that URL, it gets another vote. If it’s already on the server for another URL, those two URLs are marked as mirrors of one another.
  • If someone downloads a file others have downloaded before, and their hash does not match the hash with the most votes for that URL, it’s probably corrupt. They can use the hash tree uploaded by other successful downloaders together with download resume server features to selectively re-download just the corrupt parts.
  • Finally, even if no one else has downloaded the file, once it has been re-downloaded twice, any portions of the file which are byte-for-byte identical are probably both correct. Only portions which are different need to be re-downloaded.
  • Once at least two or three people have downloaded a file and both got the same hash, it makes sense to automatically set up a torrent for the file so that downloaders can begin to share bandwidth if they wish to, while still taking advantage of the main HTTP/FTP server (web seeding). A lot of distributions set up community torrents but don’t web seed them, leading to them becoming very slow (especially for less popular files).
Just some ideas – I’m not sure if I’ll have time to work on this, and it would require a certain critical mass of users to be effective, but I think it’d be very useful.
Posted in Uncategorized

Windows 7 hanging with black screen on boot

Written by dcoetzee on 2011-08-14 — 4 Comments

This was a frustrating problem for me: Windows installed a mandatory security update and rebooted my system without asking. Then, the system won’t start back up. I get a logo screen, then I just get a black screen, with the CPU peaked (based on fan noise) and the disk light showing little to no activity. I try to boot into Safe Mode, and it tells me it’s rolling back updates. I try to boot into Safe Mode again, and it hangs at the “Please wait” screen and makes no progress. I anticipate that a System Repair will be needed, so I decrypt my volume and then do a System Repair, but Windows says there’s nothing to repair. I try to use System Recovery and I’m informed that there are no System Recovery checkpoints.

My next step is to try to run sfc to check system files, from the command prompt in the Windows 7 repair options. I run:

sfc /scannow /offwindir=d:\windows /offbootdir=d:\

I’m informed that:

“There is a system repair pending which requires reboot to complete. Restart Windows and run sfc again.”

This isn’t very helpful, since of course starting my system is the problem. However, I discovered that by renaming d:\windows\winsxs\pending.xml, I could eliminate the pending system repair and allow the sfc run to complete.

After doing this, probably just because I renamed pending.xml, I could successfully boot into Safe Mode. Now comes part 2. I went to Device Manager and, on a hunch, went to the most recent driver I’d updated, a video driver for my NVIDIA GeForce GT 230M, the Verde version 280.26 driver released on August 9, 2011. I had installed this driver a little while back and not rebooted until now, and a black screen often points to video driver issues, and I could boot in Safe Mode which doesn’t use the video driver, so it made sense that it might be behind it. I clicked properties and did “Roll Back Driver” to get back to version 275.33 which I was using before. After doing this, and rebooting, I could boot normally again. I went ahead and installed the latest Windows updates (not including the graphics update) and rebooted without incident.

I went ahead and left a post about this at the NVIDIA forums. Hopefully the bad driver will get fixed and a new minor version released soon.

Posted in Uncategorized

Cleanly disconnecting an iSCSI disk in Windows 7

Written by dcoetzee on 2011-06-28 — Leave a Comment

At home I have a NAS with 3 disks in a RAID 5 configuration that exposes an iSCSI volume that I use from Windows machines. I chose iSCSI so I could format it with NTFS and use advanced file system features that the NAS doesn’t support natively. Although NTFS does not support multiple clients connected to the same iSCSI volume at once, if I want multiple machines to use it simultaneously, I just connect it to one machine and share it out over Windows File Sharing / Samba / SMB.

However, although Windows 7 ships with iSCSI support, it makes it awkward to transfer it from one machine to another in the LAN, which is sometimes useful. If I’ve used the disk at all, it will inevitably decline to disconnect it with the following error message:

The session cannot be logged out since a device on that session is currently being used.

Even when I’ve closed all apps that could possibly be using the disk, the message persists. But it turns out there’s a simple way around this:

  1. Open up Control Panel -> Administrative Tools -> Computer Management, and in the left pane click on Storage -> Disk Management.
  2. Right-click the iSCSI disk (the disk itself, all the way to the left, not the logical partition) and choose “Offline”. This disconnects the disk. It is not the least bit deterred by applications actively using the disk (they will receive an error if they attempt to continue reading it).
  3. Proceed to disconnect the iSCSI volume normally. It will succeed without any delay.

When later re-connecting the disk back to the original machine, you must return to Computer Management, right-click the disk, and choose Online before it is accessible again. Remember when connecting to an iSCSI disk to not add it to your “Favorite Targets” because this tends to lead to multiple machines accidentally connecting to the same iSCSI disk at the same time later on.

Posted in Uncategorized

The unconquerable “mount error 13 = Permission denied”

Written by dcoetzee on 2011-05-24 — 1 Comment

When you try to mount a Windows (SMB) share on a Linux machine using CIFS, the error reporting gives you very little insight into what could be causing the problem. This weekend, I encountered a mysterious and very persistent error “mount error 13 = Permission denied” whenever attempting to mount a share on my Windows 7 laptop. This error normally indicates that the credentials are invalid. This was strange because:

  • I was able to mount shares on another Windows 7 machine at home that was in the same workgroup and had a user account with the same username and password. (not a credentials issue)
  • The Windows 7 machine at home was able to access the shares on my laptop, and I was able to access other servers on my laptop from Linux. (not a firewall/networking issue).
  • Even a fresh VM running Ubuntu 10.10 32-bit experienced the same problem (not a Linux configuration issue, regression in Ubuntu 11.04, or 64-bit specific issue).
  • Whether the VM is run on the same machine or a different machine, whether it uses a private network or NAT or bridged connection, the same problem occurs.

I suspected a misconfiguration on my Windows 7 laptop – it was initially part of a domain, but removing it from that domain did not fix the problem. I checked the settings in the Local Group Policy Editor under Local Computer Policy -> Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> Security Options, and they matched those on the “good” machine. I also couldn’t find any information about the failed log-on attempts in the system log on either the server or client.

My solution, ultimately, was to reformat and reinstall Windows 7 on my laptop. This worked, suggesting that indeed something had got haywire with the Windows 7 configuration, although still I can’t fathom what it was. Hopefully if someone else runs into this problem they’ll come up with a better solution.

Posted in Uncategorized

Mysterious VMware shutdowns when I lock my computer

Written by dcoetzee on 2011-05-24 — 1 Comment

I have an Ubuntu 11.04 VM running in VMware on my Windows machine, and twice this morning after locking my computer and going away for a few minutes, I came back to discover the VM had powered off. At first I suspected a system crash, but there were no errors in the system logs.

It turns out that the way I was locking my computer was to press CTRL+ALT+DEL, then press ENTER to choose “Lock this computer”. In VMware 7.1.4, the CTRL+ALT+DEL keypress is also sent to the VM when it has focus. In Ubuntu 11.04, this key is bound by default to the “Log out” action, which brings up the “Log out” window. If you leave the Log out window up for 60 seconds, it automatically shuts down the system.

The solution: go in Ubuntu to System->Preferences->Keyboard shortcuts, scroll down and click on Desktop->Log out, and press backspace to remove the key binding (or select another one of your choice). Another alternative is to use WINDOWS+L to lock your computer, which should be easier anyway.

Posted in Uncategorized
« Older Entries
The author waives all rights to all content under the Creative Commons Zero Waiver (CC0 1.0 Universal).
Built on WP Framework — Powered by WordPress