FreeNAS, Linux, ZFS, and a lesson in debugging slow file transfers...

A while ago, I had my Proliant N40L server running as my home NAS, on FreeNAS 0.7, which was working rather well as a file server. Didn't have much fun getting the DLNA streaming working properly, but the ZFS implementation was great, the speeds were rapid, and it was quite brilliant. Unfortunately I also used 4x 1TB Seagate Barracuda desktop drives in it, which resulted in one dying after a while with the vibration in the cage. Oops.

So, one rebuilt server later with 4x 3TB WD Red NAS drives, I was ready to build a new tank. Decided to have a go with FreeNAS 8.2, which worked quite well. However, don't be fooled by the out-of-box configuration and built-in tweaker - I had to drop in quite a few optimisations from my old 0.7 setup to get it running quickly. But it was running quite well, it has to be said. There's 8GB of RAM in this box, for context later.

However, after some thought, I decided I wanted to get the Plex Media Server running. Sadly, at the time of writing this, it doesn't support FreeBSD, nor can FreeBSD run it under the Linux emulation as it uses epoll, which isn't supported at the moment on the BSD side. So... I decided to have a go with Linux.

One thing I didn't want to lose from the FreeNAS implementation, which was one of the primary reasons for using it, was ZFS. A while ago I'd discovered ZFS on Linux, a full kernel module for Linux, which neatly sidesteps the licensing issue (ZFS is licenced under the old Sun CDDL, which is incompatible with GPL, which is why ZFS isn't happily in the Linux kernel and doing amazing things). So, installed a copy of Ubuntu 12.04 server onto a USB flash drive via a VM on my PC, booted the microserver up off the USB key, and added the repositories for ZFS on Linux. A couple of quick installs later, and lo and behold, ZFS on Linux. Surprisingly, the performance outdid both FreeNAS implementations, although that could be other factors at work at this point:

FreeNAS 0.7

freenas:/mnt/tank# dd if=/dev/zero of=./testfile bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 87.457035 secs (122773637 bytes/sec) - 117MB/sec write
freenas:/mnt/tank# dd if=./testfile of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 60.337002 secs (177957437 bytes/sec) - 169MB/sec read

FreeNAS 0.8 (before tweaks)
[root@freenas] /mnt/storage# dd if=/dev/zero of=./testfile bs=1M count=10K && dd if=./testfile of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 225.523565 secs (47611070 bytes/sec) (45.4MB/sec)
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 78.246473 secs (137225588 bytes/sec) (131MB/sec)

FreeNAS 0.8 (after tweaks)
[root@freenas] /mnt/storage# dd if=/dev/zero of=./testfile bs=1M count=10K && dd if=./testfile of=/dev/null bs=1M && rm ./testfile
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 62.405507 secs (172058825 bytes/sec) - 164MB/sec write
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 53.998347 secs (198847165 bytes/sec) - 189MB/sec read

And finally, Ubuntu 12.04 with ZFS on Linux:

root@RickNAS:~# dd if=/dev/zero of=/storage/testfile bs=1M count=10K && dd if=/storage/testfile of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 67.544 s, 159 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 31.78 s, 338 MB/s

rick@RickNAS:~$ dd if=/dev/zero of=/storage/testfile bs=1M count=10K; dd if=/dev/zero of=/storage/testfile2 bs=1M count=10K; dd if=/storage/testfile of=/dev/null bs=1M; dd if=/storage/testfile2 of=/dev/null bs=1M

10240+0 records in

10240+0 records out

10737418240 bytes (11 GB) copied, 61.1301 s, 176 MB/s

10240+0 records in

10240+0 records out

10737418240 bytes (11 GB) copied, 59.9313 s, 179 MB/s

10240+0 records in

10240+0 records out

10737418240 bytes (11 GB) copied, 47.6692 s, 225 MB/s

10240+0 records in

10240+0 records out

10737418240 bytes (11 GB) copied, 32.2975 s, 332 MB/s

Quite strange, the improvement, but I wasn't going to knock it. So, all looked good, until I tried doing file transfers over CIFS (Windows networking), FTP and SFTP. The results were... abysmal. CIFS and FTP started fast, and rapidly dropped to about 30MB/sec. SFTP was... well, 3MB/sec didn't impress at all. To ensure that ZFS wasn't the issue, I reformatted one of the partitions as ext4, and did another test:

root@RickNAS:/storagetest# dd if=/dev/zero of=./testfile bs=1M count=10K

10240+0 records in

10240+0 records out

10737418240 bytes (11 GB) copied, 66.944 s, 160 MB/s

root@RickNAS:/storagetest# dd if=./testfile of=/dev/zero bs=1M

10240+0 records in

10240+0 records out

10737418240 bytes (11 GB) copied, 68.2629 s, 157 MB/s

root@RickNAS:/storagetest#

Not as fast as the raidz was, but still reasonably respectable. Sadly, I got similar results from CIFS etc.

Deciding to stick with ext4 for the moment just to eliminate any extra possible issues, I started to wonder why the performance was so terrible. I ran iperf to my PC, and got pretty respectable speeds out of it, far exceeding the data speeds I was getting in transfer. I went to work trying to improve the Samba speed, to no avail. No matter what I tweaked, nothing seemed to have any effect.

Next step: attempt reinstall natively, after having written out the ISO to USB.

So, reinstalled, after using the PenDrive Linux installer. Got it installed natively this time. Made zero difference. SFTP speeds are pathetic, around the 10MB/sec mark. Across gigabit, that's a bit of a joke, and it's nowhere near maxing CPU, so it's not that as far as I can tell. Did read up online that the driver for the onboard NIC can sometimes be less than perfect and to upgrade to a newer kernel. Did that. Nothing happened. Well, time passed, that's about it. iperf figured look okay though, averaging about 850mbit mark, which isn't too shabby. So there's some sort of bottleneck going on, and it's getting quite frustrating.

Next, I've tried to download /dev/zero via SFTP (in FileZilla on the Windows side) and it's settled for now at about 7.2MB/sec download speed. There's something quite clearly wrong. It's using 36% of one CPU for SSHD and 4-5% for sftp-server. It's clearly quite broken. I created a 4GB file in tmpfs in RAM for transfer tests, and vsftpd has slowed to 6.6MB/sec, which is very broken indeed. Make that 6.1MB/sec. Ouch. Rerunning the SFTP transfer resulted in SSHD taking up 96% of CPU at first (fair enough, maxed out) and doing about 40MB/sec, but it rapidly dwindled back to the 8MB/sec mark, using 26% CPU.

Next step: trying local FTP. Hooked in via localhost, retrieved said ram-based file and wrote to disk. 4294967296 bytes received in 26.49 secs (158306.3 kB/s). So, 26s to download the 4GB file, 157.5MB/sec. Seems reasonable. The put was even faster, 13.58 secs for the 4GB. So clearly it's not the software, or at least, not when running locally.

Okay. Back to the basics. Ran a few more longer iperf tests, for 6 minutes at a time each way, speeds were around 820mbit/sec on average. Nothing too spectacular, but nothing too unreasonable, and I was using the PC that they were connecting to at the time as well. Out of curiousity, I installed the CIFS utils and mounted up my Windows network share onto Linux, to copy some files across via the protocol to see what happened. This is where it got a little more confusing.

First, I ran DD to test it, using an mkv video file, 1.6GB. The second time I ran it after dumping the cache, just to make sure, was a little slower than the first time but still fairly respectable.

root@RickNAS:/mnt/win/Rick/Downloads# dd if=./testfile.mkv of=/tmpfs/testfile.mkv bs=1M

1548+1 records in
1548+1 records out
1623222074 bytes (1.6 GB) copied, 16.5486 s, 98.1 MB/s

98MB/sec. Reasonable enough, I think it came out at about 105-110 last time. The same timed operation with cp, after a reboot to flush caches, and a different file of similar size at 1.53GB (as I was struggling to kill caches):

root@RickNAS:/mnt/win/Rick/Downloads# time cp ./testfile.mkv /tmpfs/testfile.mkv
real    1m25.174s
user    0m0.096s
sys     0m6.992s

So, 85 seconds to copy 1.53GB, which equates to just under 18.5MB/sec. Seems a bit harsh. Dropped the cache, and ran it again to see what happened, after the Windows box had cached it:

real    0m17.701s

user    0m0.096s

sys     0m7.248s

So. No real problem copying there, at 88.5MB/sec. Similar to the above then, really, but a little slower. Still, acceptable enough. At this point, I have to eliminate the Windows box as a potential source of slowdown. Theoretically it shouldn't be, as it's on a RAID 0 with a pair of 500GB drives, but... one way to find out. Cue DD for Windows, and a /dev/zero device driver (translated: \\.\zero):


C:\temp>dd if=\\.\zero of=.\testfile bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 11.5837 seconds, -0.0 MB/s

Well, that -0.0 MB/s is wrong, but the theory is reasonable. Trying a 10GB file for good measure:

C:\temp>dd if=\\.\zero of=.\testfile bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 812.172 seconds, -246489356887425550000[snip] MB/s

Erm... 0.21MB/sec?! I know I did a bit of web browsing in the meantime... but there's something wrong there. I'm somewhat hoping it's the /dev/zero driver...

Okay. So plan B. Let's try a dd from the Linux box to the Windows box, the other way around.

root@RickNAS:/mnt/win/Rick/Downloads# dd if=/tmpfs/testfile.mkv of=./testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 123.167 s, 13.4 MB/s

... This would be a facepalm sort of moment then.

So what the hell is going on with the Windows box?!

CrystalDiskMark didn't report anything too strange on sequential read/write. 178.8MB/s read, 102.8MB/s write.

After this, I was entirely unable to access any of the mounted files on the Windows side, which left me rather confused. Had to drop this fix into my Windows box, and then all was shiny again. Very odd. So, another try of the DD then...


dd if=/tmpfs/testfile.mkv of=./testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 60.3011 s, 27.4 MB/s

Slightly better, but still a bit rubbish, in all honesty. Again, for good measure:

Resource monitor did show the write speed at 75MB/sec... briefly. Oh.

1576+1 records in

1576+1 records out

1653435119 bytes (1.7 GB) copied, 18.7947 s, 88.0 MB/s

... I don't get it. Same again got 87.3MB/s. It shows it as writing. Okay. Let's try another FTP download shall we? Nope, still rubbish.

Okay, time to eliminate the hard disk as an issue. Time to download to a USB HDD and see how that goes.

... Not so well. It chugged along at about 11MB/sec. That's quite poor. But it was fairly consistent, at least. CrystalDiskMark gave it a whole 10MB/sec write too, so no shock. Okay. Throwing a pure RAM disk into Windows to take out the drives as culprits.

dd:

root@RickNAS:/mnt/win# dd if=/tmpfs/testfile.mkv of=./testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 16.2628 s, 102 MB/s

Looks far more like normality.

ftp:

File transfer successful, transferred 1,653,435,119 bytes in 16 seconds.

So, that'll be... 98.55MB/sec then. Oh good.

Out of curiosity, I ran sftp, just to see what the performance was like, although I was expecting "dire". I got what I expected, about 20MB/sec. Fair enough, that's CPU-bound. Okay, other way round, just to make sure:

root@RickNAS:/mnt/win# dd if=./testfile.mkv of=/tmpfs/testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 16.1864 s, 102 MB/s

Looks good. Trying cp commands both ways:

root@RickNAS:/mnt/win# time cp /tmpfs/testfile.mkv ./testfile.mkv

real    0m16.803s
user    0m0.064s
sys     0m7.208s
root@RickNAS:/mnt/win# time cp ./testfile.mkv /tmpfs/testfile.mkv

real    0m15.218s
user    0m0.060s
sys     0m7.432s

So. RAM to RAM is actually fine. I'm going to run Samba to test it out too, to make sure all is as it should be...

... From: 100MB/sec. Copying back 110MB/sec.

Oh.

Okay, so let's try Linux HDD to ramdisk, just to make sure all is shiny as it should be.

... About 110 each way.

So. After all that hunting... all that debugging... all that chasing ghosts, upgrading network drivers and kernels, tearing my hair out... there was nothing wrong with the Linux box at all... it turned out it was my local hard disk. I just hope that this post saves someone else the hours of attempted fixes I tried. It's been a painful journey, but a lesson well learned.

Displaying Magento tier prices instead of main prices for groups

I've been tackling a particularly annoying problem with Magento, in that I've got specific pricing by groups, imported from another database, but while it'd show the tiered prices in a box, it'd always show RRP on the front page. Fixing it to show the first tier price for quantity 1 on the product page was easy enough (using $this->getTierPrices()), but the tier prices simply aren't present on the grid / listing page, and it really didn't want to let me retrieve them by group either.

So, in case it helps anyone else out at a later date, this is how I got around it, displaying tier prices instead of the main prices on the listing page as well as the main product display page. (Tested on Magento 1.4.1.1, later versions may vary.)

Around line 58, after the following

getPrice($_product, $_product->getPrice()) ?>
getPrice($_product, $_product->getPrice(), $_simplePricesTax) ?>
getPrice($_product, $_product->getFinalPrice()) ?>
getPrice($_product, $_product->getFinalPrice(), true) ?>

add:
getTierPrices();
if (empty($_tierPrices) && $this->helper('customer')->isLoggedIn()) { // No tier prices found, attempt manual retrieval if customer logged in
$resource = Mage::getSingleton('core/resource');
$query = 'SELECT qty AS price_qty,value AS price FROM '.$resource->getTableName('catalog/product').'_tier_price WHERE entity_id = '.$_id.' AND qty = 1 AND customer_group_id = '.Mage::getSingleton('customer/session')->getCustomerGroupId();
$_tierPrices = $resource->getConnection('core_read')->fetchAll($query);
}
if (!empty($_tierPrices)) {
foreach($_tierPrices AS $tierPrice) {
if ($tierPrice['price_qty'] != 1) continue;
$_price = $tierPrice['price'];
$_finalPrice = $this->helper('tax')->getPrice($_product, $tierPrice['price']);
$_finalPriceInclTax = $this->helper('tax')->getPrice($_product, $tierPrice['price'], true);
break;
}
} ?>


I also had to alter the following line lower down to avoid displaying "as low as" for items that didn't have lower than the first tier price:

From:
getDisplayMinimalPrice() && $_minimalPriceValue && $_minimalPriceValue < $_product->getFinalPrice()): ?>

To:
getDisplayMinimalPrice() && $_minimalPriceValue && $_minimalPriceValue < $_finalPrice): ?>

Hope this helps someone out, as it had me scratching my head for a bit, and the online documentation I found wasn't brilliant. In the end the most efficient way seemed to be to rip it straight out of the database. Credit to the stack overflow question that pointed me in the right direction to run the query, modified to select only the customer logged in group and quantity 1 for efficiency.

Exciting adventures with ZeroShell and Hyper-V

Perhaps "exciting" was a little overenthusiastic, but nonetheless I'm happy to publish my rebuild of ZeroShell that runs under Microsoft Hyper-V, at least on our Windows 2008 R2 Core server. This was... an interesting journey, involving a lot of kernel recompilation, patching, initrd editing and banging my head against the desk, but with a bit of shoehorning, the synthetic network adapters and the virtual storage system seem to be working quite well. I've not put it under stress testing or long-term tests yet though.

I've posted up at the ZeroShell forums with the full information on it, and would be quite happy to hear any experience anyone has with it, or if it works for them. I'm hoping to use it to do bonded VPN over our ADSL lines here on a virtualised system rather than an old PC that is currently running pfSense. Porting the firewall rules in is going to be an interesting experience, although I suspect I'll use Firewall Builder on an Ubuntu VM on my PC here, but that's going to be a story for another day.

The direct download link is here. You may need to delete /Database/var/register/system/net/interfaces/ETH* to have your network interfaces appear "normally", as I didn't clean it up before compressing it - sorry about that! More details (and no doubt feedback) will also be on the link above for the ZeroShell forums.

Windows 8 Consumer Preview: First impressions

I got my hands on the Consumer Preview of Windows 8 courtesy of Microsoft, imaged my netbook and PC, and decided initially to try a clean install on the netbook (a Dell Mini 9, 2GB RAM, 32GB SSD, 32GB SD card, 3G modem built-in, wifi/bluetooth etc, 1024x600 widescreen display). All installed fine, until rapidly realising that I couldn't actually use most of the metro apps. Then noticed I was running in 800x600, and had no display adapter installed. Fine, installed the Windows 7 driver, all great and accelerated.

... And still couldn't run the metro apps. Minimum stipulated resolution is 1024x768.

I did ugly registry hacks so they would work, via a virtual resolution scaling tweak (1024x768 into 1024x600, or 1152x864->1024x600) and they worked fantastically, albeit they looked somewhat lousy and stretched. I can understand locking a minimum resolution for games, but for the messaging app and mail clients, and Metro-ified IE? Pointless, utterly pointless, they'd have fit perfectly well. It's an arbitrary setting. The suggestion I'm going to stick on the MS feedback forums, and I hope it doesn't fall on deaf ears, is that each application should have a minimum resolution setting internally or via the registry or similar, and if it's below the minimum acceptable for that application - and to be honest, 1024x600 would likely be the bottom end, as there's a lot of the older netbooks that are at that resolution. Fine, don't give me the pinball or the option to run the apps that require a bigger screen, but don't impose that I can't run any because it doesn't fit an arbitrary minimum resolution, when they clearly would perfectly well.

After this, I wiped the netbook again, and tried doing an upgrade from the preview, as it hadn't picked up my built-in 3G mobile broadband card properly (by which I mean "it didn't show the signal like it did in Windows 7, and I couldn't do the nice automatic setup"), and I couldn't find the bluetooth system anywhere. It got all the way to the end... then told me it'd failed, and rolled it back. Nice. Installed it clean again, had a play with it, but it just seemed to be like Windows 7, with extra bits I couldn't actually use. I got severely irritated, and imaged it back to Windows 7.

Not to be deterred, I also installed it on my main desktop PC, after duly taking an image. On here, I fared a bit better - the Metro apps seemed to work. Well, they launched, anyway, configuring quite a few of them was nothing short of a nightmare. Clicking on add account to mail or messaging brought up blank boxes, attempting to add email accounts to the Metro mail client often got an immediate response of being unable to add the account or to provide further details - yet when I rebooted it worked?! The messaging client was quite annoying too - it was actually nice enough to use, but refused to actually sign out. Using the new Metro start screen wasn't too bad. As soon as you assigned Chrome as your default browser, Metro IE wouldn't work anymore (it reverted to desktop), and the app store just plain didn't work most of the time for me once I tried to clean up the duplicate machines I had due to multiple reinstalls, no matter how I tried to re-add the machine to the app store. Apparently once it's gone, it's gone, and I wasn't going to reinstall or delete my user for a few Metro apps. It also felt very weird that I couldn't just drag the start screen across with my mouse and clicking/dragging, like I would on a touchscreen - at least, I'm assuming I would. It felt completely counter-intuitive to use a scroll wheel to have to move across. Another note on counter-intuitiveness - the "charms", as they're apparently called. Top right and bottom right of the screen pull up a bar for settings and the like on the right hand side... yet I found myself continually going to the right hand side of the screen to "request" said bar. That bar should really just appear when you're on the right, if you want it to make any sense. The top left task switcher seemed like "metro app, metro app, metro app, WINDOWS DESKTOP AND ALL IT'S APPS, metro app", and the start screen was... well. It didn't make me happy. Clicking start, typing in what I'm looking for - great! You got that right, Microsoft, good show. It's what I do on Windows 7. But did you really need to throw that interface in, or is it change for the sake of it? Another bug that annoyed me - when I was networking into my NAS box, it plain refused to remember my password, regardless of clicking the "Remember credentials" button. Not sure why, could have been because it was the same name (capitalised and all) from Windows with a different password? Who knows. I'll report it anyway.

There's a lot of good in there. It's fast. From reading up, looks like they threw out some of the old compatibility crud. This is good. If people still need stuff that's just compatible with XP, I've no doubt XP mode will be made available in the same way it was for 7, and the old support for the likes of Windows 95/98 apps really needed cleaning out a long time ago. There's some nice concepts in Metro, but they need major polish, and an interface should be complementing my thought process, not making me switch mental contexts when I want to do things, and not getting in my way. That I think is my biggest complaint - Metro felt in the way, not a complimentary technology, just unnecessary cruft thrown in my face because that's the direction Microsoft want to go. Mind you, if they gave me the 7 start menu on the 8 core, I'd be happy with that, too. I can't see that happening any time soon though, it seems to be far too big a part of their strategy, and if it's not different enough, they'll not see people buying it. Of course, they could go the OS X route - "Here's your upgrade, it's not that unlike the last version but it's better, faster, and you'll appreciate the changes, £30 please" - which I'd happily pay. But again, I can't see that happening, seems they're already quite set on their course, and that would no doubt make the bean counters quite unhappy, not getting the margin they're looking for, although I suspect they'd sell a whole lot more units. Still, mere speculation, of course.

Sorry Microsoft, but unless you do something to drastically improve the experience between the Consumer Preview and the final thing, it's not something I'll be jumping on, or recommending to anyone. Which is a shame, I had a lot of hope for this operating system, and a lot of enthusiasm. Let's hope you do a better job with Windows Phone 8 - I quite fancy one of those Nokia Pureview cameras on a multi-core Windows 8 Phone. Hopefully Windows 8 apps will make it easier to get Windows 8 Phone apps too, as it's currently trounced on that score by iOS and Android, although there's obviously a lot of pointless rubbish on said app stores.

I want something that'll do everything I want, and not get in my way doing it. Do that, and you've got my custom and my heart, on the PC desktop. The phone side too for that matter. Until then, I'll be sticking to Windows 7, which does everything I need it to rather admirably. I'm wondering if Microsoft have put something that good together in Windows 7 that they're going to face the XP effect, which is still on desktops 11 years later, because it was good enough for the purpose.

Choosing a phone...

As mentioned in my previous post, I have a love/hate relationship with my Nokia N900. It's great as a computer, but not so hot as a phone, and frankly I'm sick of having to hack stuff to death to get it working in the way I want. To be honest, that's been the basis for a lot of my change in philosophies lately - I shouldn't have to hack and squeeze and force things into place to make them happen, I want things to "just work", or "just work after a bit of poking into the right shape". I'm not averse to a little bit of configuration, but I want things to work the way I want afterwards, not having to cook my own ROMs or mess with bootloaders.

The truth is I'm fed up of hacking. I'm keeping the N900, naturally, as my pocket portable computing device. I'll have a portable wifi hotspot shortly that I can use for the purpose. But for my phone... I want something reliable that works, reads my work mail properly via Exchange ActiveSync, reads my personal email properly via a push mechanism, syncs up my contacts/calendars/etc, plays music well, has a reasonable battery life, has a reasonable storage space, a hardware keyboard would be nice but I'm not averse to using a decent on-screen one if it's available, reasonable audio/video streaming from the internet, high reliability, good resilience to day-to-day usage, hooks up to my preferred social networking sites, plays games for when I'm in the mood (particularly emulated ones are of interest - I'm a sucker for Sonic the Hedgehog on the Megadrive, used to love that, although I'm happy to get ported versions if applicable), a decent web browser, decent calendar, decent camera - I want a phone with a proper camera, sick of pictures that are just "okay" off mine - decent task manager... mind you, hell, I put a list together, on my laptop, which I was checking against an iPhone in the Apple store before the Apple shop police came along and asked me ever so politely if I'd mind putting it away while I was there as they were also trying to sell computers. I hadn't even considered using pen and paper, or anything quite so low-tech, and didn't have the dropboxed file on my phone at the time.

This is what I use my phone for:

1. Sleep analyser / sleep pattern alarm (not essential, but nice)
2. Alarm
3. Streaming internet radio
4. Games (sonic the hedgehog, emulated, Megadrive, substitutions allowed)
5. Playing music
6. Satellite navigation
7. Lookup nearest [x]
8. Price comparison of products
9. Occasional wifi hotspot
10. Update FB, twitter, G+
11. Use a calculator
12. Browse the web
13. Make notes - preferably in some sort of filed fashion, unlike my random notepad files on the phone right now
14. Find contacts, browse history of communication with them (via IM/text/email/whatever) - my current phone isn't brilliant at merging the histories, but the above would be nice.
15. TV remote
16. Windows remote desktop
17. YouTube
18. Stopwatch
19. Cooking timer
20. Bandwidth monitoring
21. Logitech Squeezebox control
22. Parking car - or more precisely, finding where I parked it
23. Taking pictures
24. Taking video
25. Playing music on headphones/speakers via 3.5mm jack
26. Send/receive files via Bluetooth
27. Stream Bluetooth audio
28. Occasionally: FM receiver
29. Occasionally: FM transmitter

Would like:

1. Augmented reality stuff, like the AR satnav, and other cool apps. Purely because it's cool.
2. Ability to stream media to other devices. Basic on-device DLNA/UPnP type server would do.
3. Properly combined contact activity streams - including G+ if possible.
4. Micro-USB charging, as everything else is also moving towards using it.

I know that "TV remote" is likely a no-go on any modern phone due to the lack of infrared ports, so I'm not bothered about that. FM receiver isn't used much, as the quality tends to be quite dire. The transmitter tends to be if I'm in someone else's car, but again, it's pretty rare, and it's not like there isn't accessories about to do it - I've a standalone one floating around somewhere myself anyway, so again, non-essential. No 3.5mm jack is a dealbreaker. I also don't want to have to pay money for extra things just to be able to do tasks I can already quite happily do, if possible. I understand a small extra outlay may be required for some things, and losing the massive 32GB storage on the N900 (I say that, but I'm using about 45GB of videos, music and software on there from memory, via internal and SD card!) would hurt unless I get something roughly equivalent, but the above is my thoughts.

Since then, I've done trawling around the Trafford Centre, and worked out that visiting phone shops is, for the most part, pointless. One thing I do know - between trying to use it between cable ties - is that my initial impressions of the Nokia Lumia 800 weren't good. The browser didn't seem intuitive at all, I couldn't figure out initially how to navigate anywhere, and things seemed to keep randomly going between applications - but buttons may have been being pressed by the cable ties. One of the guys in the shop did recommend against going anywhere near the Lumia though, as apparently they've "had nothing but problems with it, and keep getting them brought back" - his suggestion was the Galaxy S-II or the HTC Sensation XL, both of which seemed reasonably good phones, and the Galaxy S-II had a nice bright display with the AMOLED kit. Again, didn't really get much of a play with them, except that they had the fairly generic Android experience (well, HTC Sense on the HTC, and whatever the Samsung front-end is). Not much of a fair trial though. I also managed to get shouted at in the Apple store for having my netbook out and making notes on it on the counter while attempting to try out an iPhone 4S - only to discover it was crippled beyond belief, and you got what you were looking at. It did seem to scream the whole Apple philosophy - "you can do what you want as long as it's what we want" - although I suppose I can understand them not wanting to see competitor's tech in there (the wording, which was asked ever so nicely, was "would you mind putting that away, as we're also selling computers in here." He was ever so nice about it, but it completely took me off guard, I hadn't even considered any other method of taking notes). Yet they weren't bothered about my phone. *shrug* Did kill off any interest I had of being in there from that moment on though. However, the overwhelming conclusion of that was that apparently people want to look at phones in phone shops and go "ooh, that's shiny" or "Oh, that comes in pink, I like that!", rather than actually properly road-test phones. Perhaps I should have made an appointment like I was planning on doing in order to properly test one out from scratch - but I may still be able to with a few phones, more to follow on that one. I'm not allowing my phone-shop experience to influence my end decision, as, in all honesty, the only real thing I experienced was an in-shop Lumia playing up.

So far the shortlist looks a bit like this:

• The Lumia 800 (or any Windows Phone around the same sort of spec, really - annoyingly all of which come without hardware keyboards, unless I go back a generation to the likes of the HTC 7 Pro) - which also has the advantage like all Windows Phones that I'll be able to throw code at it once I'm up to speed with C#
• An iPhone 4S - until I properly try one, I've no idea if it's what I'd actually want, no chance of getting code on it though unless I decide to learn Objective C or involve the likes of MonoTouch
• A high-end Android phone e.g. the Galaxy S-II (again, same problem with lack of keyboards, although the previous-generation spec Sony Xperia Pro is a reasonable contender, albeit far from cutting edge... but it is also £200, a third to a quarter of the costs of the above), and again, no code without the likes of Mono for Android

I did also consider some sort of barely-above-dumbphone and a tablet, or similar, but it'd probably just annoy me quite quickly.

So. Next, I need to road-test the above, and work out how they'll fit into my digital life. The one thing whichever it is needs to be able to make me happy, not irritated, and be a lubricant in the chafing of life, rather than something abrasive, which I have to fight every step of the way to achieve what I want. I'm tired of fighting with technology, it should be working for me, not the other way around.

The Status Quo

So, here's where I'm at right now technology wise.

• Computing
• 1x PC. It's not a bad bit of kit. Tech specs - 2x500GB SATA hard disks in a RAID 0, with a Crucial 128GB Synapse SSD in place as a cache drive, running Windows 7 Home Premium. Intel Q6600 2.4GHz quad core CPU with a Tuniq Tower on it, which is nice and quiet, 4GB of OCZ DDR2 PC-1066 RAM, and an XFX Radeon 6870 1GB GPU. There's also a 160GB SATA hard disk that currently doesn't serve much of a purpose other than a scratch drive. Also got a Wacom Graphire4 attached, nice graphics tablet, shame it's so underused. Also got a nice BenQ FP241W 24" monitor, and an old Relisys 19" one that's dying somewhat and has a grey line running halfway across the screen about an inch down, but it's soldiering on. I'll replace it at some point no doubt. Also got a Logitech 5.1 speaker setup connected, which does well for games and video, Microsoft keyboard and mouse, cheap webcam, etc.
• 1x Printer - HP Photosmart B110. Does the job, cheap and cheerful, eats ink when printing photos, seems to jam up with some photo paper and slant the last part, making some nasty clicking sounds. Hmm. Scans reasonably well though.
• 1x Netbook: Dell Mini 9, with 2GB RAM, 32GB SSD, 32GB class 10 SD card, 3G card built-in, and the rest as stock. Originally bought for it's fantastic ability to run OS X natively. Got bored of that since, and put Windows 7 Home Premium on that too. Works pretty well.
• 1x Laptop: an old Toshiba Satellite Pro A40 from memory. Sits on my bedroom next to my bed, and mainly gets used as something to watch videos on, and the odd bit of surfing in bed. Sits on an Ikea Dave laptop table, which is brilliant. The graphics are a bit old though, Intel 82855, and it's a bit slow. Runs XP. Isn't bad at playing normal videos although I sometimes have to kill off Microsoft Security Essentials to stop it being chuggy and trying to scan things when playing stuff back. iPlayer chokes occasionally. Runs on wifi.
• 2x HP ProLiant N40L MicroServer. £240 each at the time, with £100 mail-in rebate, so £140 a pop. Each came with a 250GB SATA drive, the case has 4 caddies for drives, comes with 2GB RAM by default, an internal USB slot, but apparently they want you to purchase a drive kit if you want an optical drive (probably just needs some of those big round screws to slot in that you find on many HP/Compaq PCs). I therefore of course just installed from USB. Got an 8GB Crucial ECC DDR3 RAM kit (2x4GB) for £50, and promptly swapped the 2GB and sold-with 250GB SATA drive from one into the other, and put the 8GB in the one I just took it from, along with 4x1TB drives that I've acquired along the way, and installed FreeNAS 7.1 on it, running with ZFS, installed on a 4GB USB flash drive mounted internally. Few hiccups along the initial install (ZFS memory requirements can be a bit funky, needed a bit of tweaking, as it happily eats whatever it can in cache), but it's running like a champ now. The other I still haven't deployed, but it's got the 2x 250GB drives and 4GB cache. Might virtualise stuff on it, although the original plan was to run something like LinuxMCE on it, using it as the control server. That's not off the table yet, depends on how my experiments go.
• 4x O2 Joggler - intended for use with the LinuxMCE kit, as you can readily boot them off USB and run Linux. For the £50 a pop they were at the time, plus cashback, they were a bargain. They've actually ended up mostly running as Logitech SqueezeBox clients though. Just need to get that server fixed up on the NAS box... but that's another story for another post.
• Car PC to be: An Asus Eeebox that runs on 12V, 2GB RAM, 250GB or so hard disk from memory, and a Lilliput LED-backlit touchscreen. More to follow on that one, but it'd be nice if it all integrated nicely with the stuff at home.
• Bunch of old hardware in the graveyard, which I really should do something with. Like selling on eBay, perhaps.
• Networking
• ADSL Router is a BeBox, aka a Thomson Speedtouch 540. It works well enough bridged straight into my main router.
• 2x Linksys WRT54GL routers, running Tomato. Works *extremely* well. Got the VPN-enabled version on them, which also comes with full bandwidth limitation on it. One connected directly to the router via the above bridge, and one in the front room. Total overkill I suspect, but never mind. The second is connected via a basic cheap 100mbps powerline ethernet link, via the gigabit switch between them below.
• 1x ZyXEL GS-108B  8 port gigabit switch. Cheap, cheerful, does a fine job.
• Home cinema kit
• Receiver - Denon AVR-1910. Works well enough, but annoyingly the dynamic EQ stuff would never quite work in my room, and couldn't figure out where the sound was coming from with regards to the speakers. Always bugged me.
• Speakers
• KEF 2005.3 5.1 kit, used as front centre, two sides and two rears on stands, and a KEF KUBE 2 that came with it.
• KEF IQ... something... floor standers. Really can't remember. They're tall, and sound nice? IQ5 maybe.
• Playstation 3. It plays the videos and blu-rays, and games, naturally.
• Wii. Not strictly home cinema, but it's connected up, and runs games from the hard drive using WiiFlow. PS2 as well running games from hard disk using Free McBoot and Open PS2 Loader.
• LG CF181D LCoS projector. Brilliant, brilliant bit of kit. Shame I can't zoom out enough to have it at one side of the room with the screen at the other! Admittedly took me about 6 months to get the thing sorted properly as the first one that arrived didn't work properly after 15 minutes of use, but worth the wait.
• Beamax motorised remote control screen - comes up from the floor. Again, pretty damn cool, and semi-portable (as in portable like luggable laptops were). Tends to end up staying up most of the time, which probably isn't great for it. Must oil those cables.
• Mobile
• Nokia N900. Ahh, the best I've had, the worst I've had. Fantastic mobile computer. Runs Maemo Linux, which has now been utterly hacked to death and demonstratively improved by the community (who needs a stock kernel anyway?). It's got a good web browser, Opera Mobile works fantastically on it, plays a wicked game of Sonic the Hedgehog on a Megadrive emulator, my typing speed on it is great, the camera is... okay, it's got shell access, loads and loads of Linux applications... but that's the thing. It's a great computer. It's SMS implementation is... buggy, at best (read: keeps doing stupid things with messages making me delete certain ones in order to make it work properly again - which it'll only let me delete under certain circumstances), the email was quite prone to crashing (although the community software updates seem to have taken care of that, since Nokia abandoned it), I can crack WEP networks within minutes on it and have a good go at WPA too... but Nokia walked out on it's new killer platform, and embraced Windows Phone. This phone I saw as the geek dream phone. And it was. It's brilliant. It's utterly brilliant. But sometimes I'd just like to be able to get commercial software on it other than Sygic satellite navigation and Angry Birds levels. Annoyingly, there's actually Android players for it which will run Android apps at native speeds - but they were only ever offered to OEMs, and not the end users. It sort-of runs Android as well - by which I mean you can dual boot, and quite a few functions work, which would probably be more useful if it could send/receive phone calls - specifically the audio on them. Don't get me wrong, hats off to the NITDroid guys, but yet again it's an example of stuff being hacked onto hardware that doesn't quite work. It's not their fault, they don't have access to the specs/hardware/etc, but the end user experience ends up lacking. And I'm sick of having half-arsed solutions. Oh, and finally - this phone, when trying to answer a phone call, has an annoying habit of moving the on-screen answer button when you pick it up (rotating portrait/landscape), and if you lock it to landscape, it doesn't display properly. No doubt it'll get fixed, but still. Grr.
• Camera
• Canon 550D SLR. Great bit of kit, likely somewhat wasted on me in all honesty. I love the camera, and am still learning how to use it, along with getting the right effects, and post-processing, but I haven't had time to indulge in it, unfortunately. I'm hoping that'll change soon. I've also got an SD card with the Magic Lantern firmware on it, which looks cool, if I ever end up filming anything.

So, there we have the beginning hardware. At the moment it's a jiggly mess playing stuff on the PS3 as I'm doing it temporarily via the PC on PS3 Media Server serving from the NAS box, but that somewhat defeats the point in having the powerful NAS that can run it itself (once I sort the binaries out on there, just haven't had time to fix it up). I need to transfer 64-bit binaries for the Squeezebox Server onto the FreeNAS box to get the audio all synced up etc. And once I've got all that figured out, I'll work out what on earth my plan is to make this lot converge. I also want to get home automation in, although that may have to partially wait until I have my own place, so I can drop in cat5 all over the place, and just use powerline control for it all, or RF. Time will tell. I wanted to record where I was at before I started though, so I can observe the changes as I go.

Curiously enough, the only thing that went through the mind of the bowl of petunias as it fell was "Oh no, not again."

So. Here we are again. Well, I say again, it's the first time I've been "here", so to speak, but I'm actually writing up a blog post, for the first time in $deity->knows->when(). And I've started with a HHGTTG quote for my first mostly unimportant post, as all blogs should probably begin with. A bit of fluff to start with, shall we say.

Mind you, things have come a long way. LiveJournal is all but dead except for all the Russian people. I've nothing against Russians of course, they're lovely people - well, they've got a spread of people lovely to lousy just like everywhere else has really - but unfortunately, being unable to communicate in their language, it does put a bit of a barrier up for me. The communities are all but dead too, sadly, replaced with the likes of Facebook and Google+ (well, okay, perhaps the latter is wishful thinking, but it's still very early days. The release of G+ without suitable APIs - even in alpha format - was a bit ridiculous in my opinion, and ensured Facebook was just going to sit as the dominant social network. Who knows what the future holds on that one, but... we shall see) for the most part. And, to be honest, I did question the wisdom of starting a blog when G+ and Facebook (and even Twitter) are perfectly good platforms for writing things up. Perhaps it's because I want things here to be taken a little more seriously than the irrelevant day-to-day drivel and rants I put up there, or very short real-time posts.

As an update to this post, I used to go out of my way to set up my own local copies of the blogs, organise the databases, get all the files in place, get all the plug-ins there, make it all work, fix the inevitable permissions issues, etc. But I've realised it really isn't worth the time or the effort when a perfectly good publishing platform is already available to me. Same with my mail now running on Google Apps, and generally wanting a nice easy life. Sure, Google gets to know about what I write about, the sort of people who visit my blog, the email I get, and use all of the above to customise my search experience and adverts served. That's okay, I can live with that, I usually find what I want, and it's a price I'm willing to pay. I've had enough of taking the approach of fashioning tools and then building my own house, or even just getting pre-made tools and still building my own house, or at the very least severely remodelling. I'm happy enough putting a bit of paint here and there, and connecting up bits and pieces. It's good enough, and it's a smooth enough experience to "just work" for me.

So here I shall begin my journey. My journey into a synergistic technological life. My battle to make it all work smoothly together, seamlessly integrated. The changes in attitudes I've had, and the journey I make, the tools I use to get there. My experiences in programming, with operating systems, and hardware. And perhaps exploring bits of the inside of my thought processes a little too.

So, in that spirit: are you sitting comfortably? Then I'll begin.