Please note: Since I wrote this, I started having issues again. I'll update as I find out more.
I trialled a Moto 360 when they were on offer earlier in the year, and while Android Wear was okay in itself, I wasn't too impressed with the shape, but more importantly, whenever I had the Wear application on my phone, it utterly murdered my battery. Weirdly, it was Hangouts that was keeping one core at 100% all the time, which I found quite odd, but never managed to find an explanation, or get to the bottom of it, although I did submit some bug reports.
Fast forward to now, and I'm back on Android Wear, having accidentally smashed the glass on my SmartWatch 2. And lo and behold, my phone battery was being eaten for breakfast, again, and yet again it was Hangouts.
Much experimentation later, and I finally realised the problem - I had too many active hangouts. Being an Ingress player, this is nothing new, I'm used to having 100+ on the go, but all those hangouts with individuals and groups took their toll when trying to sync. My guess is that the Wear application was trying to sync the huge amount of data to the watch, and kept failing, and kept retrying. I archived off all my non-active conversations. Boom, watch syncs up without falling over and eating my battery on the phone. Victory!
Hope this helps other people from scratching their head as to why their battery life sucks after getting an Android Wear watch, and if they've spotted Hangouts taking up a crazy amount of CPU. Once the initial sync is complete, it seems to have behaved since, but time will tell if that stays the case.
Do let me know if this has affected anyone else out there, or if it's just me...
Edit: Unfortunately, it appears that it was actually the act of clearing the data that solved the problem. After a while, it seems to get stuck again and constantly try to resync with the phone. It's rapidly putting me off using Hangouts altogether, which is a bit of an issue when most of my Ingress game communication is done on there!
Noted another thread relating to this:
https://www.reddit.com/r/AndroidWear/comments/49u6m4/high_hangouts_cpu_usage_since_getting_watch/
Will add more as I find them. This is maddening, there seems to be no source of information on it. Just installed Messenger and disabled Hangouts SMS integration, let's see how we go...
24 October 2015
Enabling BCM43142 Bluetooth on Linux Mint 17.2 and Ubuntu 14.04 LTS (Trusty Tahr)
Recently, I acquired a Thinkpad Edge E545, with the Broadcom hybrid WiFi/Bluetooth mini PCIe card (WiFi reads as wlan0: Broadcom BCM4365 802.11 Hybrid Wireless Controller 6.30.223.248 (r487574), bluetooth reads as usb 4-1: Product: BCM43142A0, USB ID is 105b:e065). The wifi worked fine as soon as I enabled the proprietary drivers in the driver manager, but the bluetooth was having none of it. Cue much frustration.
Based on the instructions and patch at this blog post for getting it working on Debian Jessie, I started to have a go, but I couldn't seem to download the kernel source - the apt-get source commands were pointing at nonexistent releases. Unfortunately, that was down to me failing to add the deb-src lines to apt-get. Once I did that, and actually followed the instructions properly, it sort of worked. But then it still kept downloading newer kernels, which wasn't overly helpful when I just wanted to recompile a module to add to my existing system.
I manually cloned the script, and adapted it to work with the Ubuntu sources from git, with the exact kernel I'm using. I've uploaded a copy of the script plus the patchfile that I modified and set up to automatically clone the Ubuntu kernel version that Mint is using from the LTS repo. Feel free to download it and modify as necessary for kernel versions, I just did a quick and dirty job. It downloads quite a bit of code, as it grabs the whole repo, and it's likely reasonably easy to do it more efficiently, but it did work. As described on the previous blogpost, you'll need hex2hcd compiled, base-devel and git installed, and the hex binary file from a Windows install, e.g. BCM43142A0_001.001.011.0311.0312.hex, which I used.
But, as I'm a kind soul, I've precompiled them, and they're working on my E545. I'm not 100% sure on the legality with the Broadcom binary files, but as they distribute the blobs for Linux anyway, I doubt they'll mind so much. This also builds on the work on the aforementioned blog post, with the script adapted from theirs. This may not work for other USB IDs - you'd need to modify the above script / patch patch appropriately. I hope this saves other people the hassle I went through figuring it out! If you've an aversion to running precompiled binaries from some random site, you're welcome to download the above script, inspect it, and run it to build your own copy - the hashes should match, in theory, with default compile flags.
The binary download for the patched btusb.ko and the firmware (for 14.04 based distros) are here. Run the shell script to install it. Can't guarantee it'll work with your exact configuration, but the script above might it this doesn't. Kernel version was a 3.16 version from memory.
Hope this saves people some hassle, please do drop a comment on the blog if it sorts you out. Good to know if it's saved people some pain. It's the exact binary that made the E545 hardware work.
10 September 2013
Magento dataflow batch exports and parent SKUs of configurable products
At work, I've been scratching my head how to easily batch export and import Magento products. Importing is covered pretty well by <a href="http://sourceforge.net/apps/mediawiki/magmi/index.php?title=Magmi_Wiki">Magmi</a>, but the exports are a bit lacking. Found an extension to add extra abilities to the Dataflow export, including category exports, in here - http://www.magentocommerce.com/boards/viewthread/60113/ - but the one thing I wanted was to be able to export the parent SKUs of complex products, as Magmi can handle them in their own field (simple_skus). However, I then realised I'd done it completely the wrong way around, and I actually needed child products. So I've extended the extension, and thrown it up on GitHub, hopefully it'll save someone else the hassle for either purpose. Seems to work well, but let me know if there's any issues, I'll see what I can do to help.
Usage is the same as the original extension, with the addition of a few options at the bottom:
<action method="unparse" type="exportextension/modifier">
<var name="remove_line_breaks">true</var>
<var name="remove_html_tags">true</var>
<var name="add_categories">true</var>
<var name="category_field_name">category</var>
<var name="category_delimiter">#</var>
<var name="category_first_level">-1</var>
<var name="category_path_delimiter"><![CDATA[>]]></var>
<var name="add_absolute_url_to_field">product_url</var>
<var name="add_image_url_to_field">image_url</var>
<var name="add_parent_sku">parent_sku</var>
<var name="add_child_sku">child_skus</var>
<var name="child_sku_delimiter">,</var>
<var name="add_child_sku">child_skus</var>
<var name="child_sku_delimiter">,</var>
</action>
- remove_line_breaks: if the value of this tag is “true”, linebreaks will be removed from the content
- remove_html_tags: if the value of this tag is “true”, html tags will be removed from the content
- add_categories: if the value of this tag is “true”, each product will be extended with category names.
The following tags, are only used, if “add_categories” is “true”. They configure the additional category field:
- category_field_name: this value defines the used name of the category field. The default value of this tag is “category”. This tag is only expedient, if you export the fieldnames aor if you allready have another field with the defaultname “category”.
- category_delimiter: if a product is at more than one categories, each category will added and they will be delimited by this string. The default value is “#”.
- category_first_level: this value defines the level of the first exportable category. If it is 2 or greater, the first category names of the category tree won’t be added to the data. The values 1 and 0 are the same, because there is no level 0 - the root category has level 1. If the value is set to “-1”, only the proper category name will be added to the product. The default value of this tag is “1”.
- category_path_delimiter: if the category paths are exported (category_first_level is not -1), each category at the path will be delimited by this string. The default value is “>”. Example: catA1>catA2#catB1>catB2>catB3
- add_categories: if the value of this tag is “true”, each product will be extended with category names.
The following tags, are only used, if “add_categories” is “true”. They configure the additional category field:
- category_field_name: this value defines the used name of the category field. The default value of this tag is “category”. This tag is only expedient, if you export the fieldnames aor if you allready have another field with the defaultname “category”.
- category_delimiter: if a product is at more than one categories, each category will added and they will be delimited by this string. The default value is “#”.
- category_first_level: this value defines the level of the first exportable category. If it is 2 or greater, the first category names of the category tree won’t be added to the data. The values 1 and 0 are the same, because there is no level 0 - the root category has level 1. If the value is set to “-1”, only the proper category name will be added to the product. The default value of this tag is “1”.
- category_path_delimiter: if the category paths are exported (category_first_level is not -1), each category at the path will be delimited by this string. The default value is “>”. Example: catA1>catA2#catB1>catB2>catB3
- add_absolute_url_to_field; wit this tag you can set a field name, which will be used to add the absolute product link. if this field does not exists, it will be added. if it exists, it will be overwritten.
- add_image_url_to_field; this tag works at the same way, but exports the absolute image link.
- add_image_url_to_field; this tag works at the same way, but exports the absolute image link.
- add_parent_sku: If this field is added, the parent SKU of any simple products associated with a configurable product will be added as a field.
- add_child_sku: If this field is added, the child SKUs of any configurable products will be added as a field. If intended for use with Magmi, this should be named as simple_skus.
- child_sku_delimiter: Delimiter for the add_child_sku field, default if unspecified is a comma.
Limitations: This will only currently pull up the first parent SKU of a simple product, if it is assigned to multiple configurables. I didn't look into that any further at the time. This does not affect child products of configurables, which will be properly seperated by the delimiter specified.
I've uploaded the source to GitHub,at https://github.com/microchip/Magento-ExportExtension - both the main extension code, and a prepackaged version in the package folder. Hope it helps save someone the pain I went through trying to figure this damn thing out. The only answer you seem to find online is "Well, if you're struggling with this, here's an extension we'll sell you to fix the issue!" Open source at its best, huh, guys? Don't get me wrong, I understand the desire to write extensions and make money, but the amount people seem to charge is a bit ridiculous, especially if you only want one feature out of the 10 the extensions that are being peddled do.
Anyway. Rant over...
- add_child_sku: If this field is added, the child SKUs of any configurable products will be added as a field. If intended for use with Magmi, this should be named as simple_skus.
- child_sku_delimiter: Delimiter for the add_child_sku field, default if unspecified is a comma.
Limitations: This will only currently pull up the first parent SKU of a simple product, if it is assigned to multiple configurables. I didn't look into that any further at the time. This does not affect child products of configurables, which will be properly seperated by the delimiter specified.
I've uploaded the source to GitHub,at https://github.com/microchip/Magento-ExportExtension - both the main extension code, and a prepackaged version in the package folder. Hope it helps save someone the pain I went through trying to figure this damn thing out. The only answer you seem to find online is "Well, if you're struggling with this, here's an extension we'll sell you to fix the issue!" Open source at its best, huh, guys? Don't get me wrong, I understand the desire to write extensions and make money, but the amount people seem to charge is a bit ridiculous, especially if you only want one feature out of the 10 the extensions that are being peddled do.
Anyway. Rant over...
02 December 2012
Solution to audio on the Toshiba Satellite Pro P100 and Windows 7 64-bit muted on install
I've just had a seriously non-fun few hours arguing with drivers on the Toshiba Satellite Pro P100 (specifically the PSPAEE-00E006EN variant). Had no real issues installing Windows 7 64 bit, didn't really expect to as Vista 32-bit worked perfectly well. 135 updates installed later, including the graphics driver, everything seemed to be working swimmingly.
Except the sound, which while it pretended to play, it clearly was not.
Apparently I was far from alone in this issue. Many forums complained about lack of Vista x64 / Windows 7 x64 failing to work properly with the sound. I tried installing a variety of sound drivers, none of which seemed to actually work, although strangely occasionally I'd try to install one, it'd fail, but the sound would work until I rebooted. Extremely frustrating.
Eventually, after much hunting down and forum crawling, I found this driver from HP - SP40170 - which did actually work. The automatic install program didn't, but installing the driver manually via device manager - update driver - select from a location on disk, and choosing c:\SWSETUP\SP40170\Venice\V64 allowed me to install a working driver, and get sound.
It's not perfect - the sound stops working on suspending the machine, which I haven't figured a solution for yet aside from reboot - but other than that it seems to be working well so far. I'm a bit surprised (and disappointed) that a proper 64-bit driver wasn't released for Vista/7 by Toshiba, but hey, just the way the cookie crumbles with older hardware I guess.
Hope this saves someone else the hours I've had to spend hunting this down.
Except the sound, which while it pretended to play, it clearly was not.
Apparently I was far from alone in this issue. Many forums complained about lack of Vista x64 / Windows 7 x64 failing to work properly with the sound. I tried installing a variety of sound drivers, none of which seemed to actually work, although strangely occasionally I'd try to install one, it'd fail, but the sound would work until I rebooted. Extremely frustrating.
Eventually, after much hunting down and forum crawling, I found this driver from HP - SP40170 - which did actually work. The automatic install program didn't, but installing the driver manually via device manager - update driver - select from a location on disk, and choosing c:\SWSETUP\SP40170\Venice\V64 allowed me to install a working driver, and get sound.
It's not perfect - the sound stops working on suspending the machine, which I haven't figured a solution for yet aside from reboot - but other than that it seems to be working well so far. I'm a bit surprised (and disappointed) that a proper 64-bit driver wasn't released for Vista/7 by Toshiba, but hey, just the way the cookie crumbles with older hardware I guess.
Hope this saves someone else the hours I've had to spend hunting this down.
12 August 2012
FreeNAS, Linux, ZFS, and a lesson in debugging slow file transfers...
A while ago, I had my Proliant N40L server running as my home NAS, on FreeNAS 0.7, which was working rather well as a file server. Didn't have much fun getting the DLNA streaming working properly, but the ZFS implementation was great, the speeds were rapid, and it was quite brilliant. Unfortunately I also used 4x 1TB Seagate Barracuda desktop drives in it, which resulted in one dying after a while with the vibration in the cage. Oops.
So, one rebuilt server later with 4x 3TB WD Red NAS drives, I was ready to build a new tank. Decided to have a go with FreeNAS 8.2, which worked quite well. However, don't be fooled by the out-of-box configuration and built-in tweaker - I had to drop in quite a few optimisations from my old 0.7 setup to get it running quickly. But it was running quite well, it has to be said. There's 8GB of RAM in this box, for context later.
However, after some thought, I decided I wanted to get the Plex Media Server running. Sadly, at the time of writing this, it doesn't support FreeBSD, nor can FreeBSD run it under the Linux emulation as it uses epoll, which isn't supported at the moment on the BSD side. So... I decided to have a go with Linux.
One thing I didn't want to lose from the FreeNAS implementation, which was one of the primary reasons for using it, was ZFS. A while ago I'd discovered ZFS on Linux, a full kernel module for Linux, which neatly sidesteps the licensing issue (ZFS is licenced under the old Sun CDDL, which is incompatible with GPL, which is why ZFS isn't happily in the Linux kernel and doing amazing things). So, installed a copy of Ubuntu 12.04 server onto a USB flash drive via a VM on my PC, booted the microserver up off the USB key, and added the repositories for ZFS on Linux. A couple of quick installs later, and lo and behold, ZFS on Linux. Surprisingly, the performance outdid both FreeNAS implementations, although that could be other factors at work at this point:
FreeNAS 0.7
freenas:/mnt/tank# dd if=/dev/zero of=./testfile bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 87.457035 secs (122773637 bytes/sec) - 117MB/sec write
freenas:/mnt/tank# dd if=./testfile of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 60.337002 secs (177957437 bytes/sec) - 169MB/sec read
FreeNAS 0.8 (before tweaks)
[root@freenas] /mnt/storage# dd if=/dev/zero of=./testfile bs=1M count=10K && dd if=./testfile of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 225.523565 secs (47611070 bytes/sec) (45.4MB/sec)
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 78.246473 secs (137225588 bytes/sec) (131MB/sec)
FreeNAS 0.8 (after tweaks)
[root@freenas] /mnt/storage# dd if=/dev/zero of=./testfile bs=1M count=10K && dd if=./testfile of=/dev/null bs=1M && rm ./testfile
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 62.405507 secs (172058825 bytes/sec) - 164MB/sec write
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 53.998347 secs (198847165 bytes/sec) - 189MB/sec read
And finally, Ubuntu 12.04 with ZFS on Linux:
root@RickNAS:~# dd if=/dev/zero of=/storage/testfile bs=1M count=10K && dd if=/storage/testfile of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 67.544 s, 159 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 31.78 s, 338 MB/s
Next, I've tried to download /dev/zero via SFTP (in FileZilla on the Windows side) and it's settled for now at about 7.2MB/sec download speed. There's something quite clearly wrong. It's using 36% of one CPU for SSHD and 4-5% for sftp-server. It's clearly quite broken. I created a 4GB file in tmpfs in RAM for transfer tests, and vsftpd has slowed to 6.6MB/sec, which is very broken indeed. Make that 6.1MB/sec. Ouch. Rerunning the SFTP transfer resulted in SSHD taking up 96% of CPU at first (fair enough, maxed out) and doing about 40MB/sec, but it rapidly dwindled back to the 8MB/sec mark, using 26% CPU.
Next step: trying local FTP. Hooked in via localhost, retrieved said ram-based file and wrote to disk. 4294967296 bytes received in 26.49 secs (158306.3 kB/s). So, 26s to download the 4GB file, 157.5MB/sec. Seems reasonable. The put was even faster, 13.58 secs for the 4GB. So clearly it's not the software, or at least, not when running locally.
Okay. Back to the basics. Ran a few more longer iperf tests, for 6 minutes at a time each way, speeds were around 820mbit/sec on average. Nothing too spectacular, but nothing too unreasonable, and I was using the PC that they were connecting to at the time as well. Out of curiousity, I installed the CIFS utils and mounted up my Windows network share onto Linux, to copy some files across via the protocol to see what happened. This is where it got a little more confusing.
First, I ran DD to test it, using an mkv video file, 1.6GB. The second time I ran it after dumping the cache, just to make sure, was a little slower than the first time but still fairly respectable.
root@RickNAS:/mnt/win/Rick/Downloads# dd if=./testfile.mkv of=/tmpfs/testfile.mkv bs=1M
1548+1 records in
1548+1 records out
1623222074 bytes (1.6 GB) copied, 16.5486 s, 98.1 MB/s
root@RickNAS:/mnt/win/Rick/Downloads# time cp ./testfile.mkv /tmpfs/testfile.mkv
real 1m25.174s
user 0m0.096s
sys 0m6.992s
So. No real problem copying there, at 88.5MB/sec. Similar to the above then, really, but a little slower. Still, acceptable enough. At this point, I have to eliminate the Windows box as a potential source of slowdown. Theoretically it shouldn't be, as it's on a RAID 0 with a pair of 500GB drives, but... one way to find out. Cue DD for Windows, and a /dev/zero device driver (translated: \\.\zero):
C:\temp>dd if=\\.\zero of=.\testfile bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 11.5837 seconds, -0.0 MB/s
Well, that -0.0 MB/s is wrong, but the theory is reasonable. Trying a 10GB file for good measure:
C:\temp>dd if=\\.\zero of=.\testfile bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 812.172 seconds, -246489356887425550000[snip] MB/s
Erm... 0.21MB/sec?! I know I did a bit of web browsing in the meantime... but there's something wrong there. I'm somewhat hoping it's the /dev/zero driver...
Okay. So plan B. Let's try a dd from the Linux box to the Windows box, the other way around.
root@RickNAS:/mnt/win/Rick/Downloads# dd if=/tmpfs/testfile.mkv of=./testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 123.167 s, 13.4 MB/s
... This would be a facepalm sort of moment then.
So what the hell is going on with the Windows box?!
CrystalDiskMark didn't report anything too strange on sequential read/write. 178.8MB/s read, 102.8MB/s write.
After this, I was entirely unable to access any of the mounted files on the Windows side, which left me rather confused. Had to drop this fix into my Windows box, and then all was shiny again. Very odd. So, another try of the DD then...
dd if=/tmpfs/testfile.mkv of=./testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 60.3011 s, 27.4 MB/s
root@RickNAS:/mnt/win# dd if=./testfile.mkv of=/tmpfs/testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 16.1864 s, 102 MB/s
Looks good. Trying cp commands both ways:
root@RickNAS:/mnt/win# time cp /tmpfs/testfile.mkv ./testfile.mkv
real 0m16.803s
user 0m0.064s
sys 0m7.208s
root@RickNAS:/mnt/win# time cp ./testfile.mkv /tmpfs/testfile.mkv
real 0m15.218s
user 0m0.060s
sys 0m7.432s
So. RAM to RAM is actually fine. I'm going to run Samba to test it out too, to make sure all is as it should be...
... From: 100MB/sec. Copying back 110MB/sec.
Oh.
Okay, so let's try Linux HDD to ramdisk, just to make sure all is shiny as it should be.
... About 110 each way.
So. After all that hunting... all that debugging... all that chasing ghosts, upgrading network drivers and kernels, tearing my hair out... there was nothing wrong with the Linux box at all... it turned out it was my local hard disk. I just hope that this post saves someone else the hours of attempted fixes I tried. It's been a painful journey, but a lesson well learned.
So, one rebuilt server later with 4x 3TB WD Red NAS drives, I was ready to build a new tank. Decided to have a go with FreeNAS 8.2, which worked quite well. However, don't be fooled by the out-of-box configuration and built-in tweaker - I had to drop in quite a few optimisations from my old 0.7 setup to get it running quickly. But it was running quite well, it has to be said. There's 8GB of RAM in this box, for context later.
However, after some thought, I decided I wanted to get the Plex Media Server running. Sadly, at the time of writing this, it doesn't support FreeBSD, nor can FreeBSD run it under the Linux emulation as it uses epoll, which isn't supported at the moment on the BSD side. So... I decided to have a go with Linux.
One thing I didn't want to lose from the FreeNAS implementation, which was one of the primary reasons for using it, was ZFS. A while ago I'd discovered ZFS on Linux, a full kernel module for Linux, which neatly sidesteps the licensing issue (ZFS is licenced under the old Sun CDDL, which is incompatible with GPL, which is why ZFS isn't happily in the Linux kernel and doing amazing things). So, installed a copy of Ubuntu 12.04 server onto a USB flash drive via a VM on my PC, booted the microserver up off the USB key, and added the repositories for ZFS on Linux. A couple of quick installs later, and lo and behold, ZFS on Linux. Surprisingly, the performance outdid both FreeNAS implementations, although that could be other factors at work at this point:
FreeNAS 0.7
freenas:/mnt/tank# dd if=/dev/zero of=./testfile bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 87.457035 secs (122773637 bytes/sec) - 117MB/sec write
freenas:/mnt/tank# dd if=./testfile of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 60.337002 secs (177957437 bytes/sec) - 169MB/sec read
FreeNAS 0.8 (before tweaks)
[root@freenas] /mnt/storage# dd if=/dev/zero of=./testfile bs=1M count=10K && dd if=./testfile of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 225.523565 secs (47611070 bytes/sec) (45.4MB/sec)
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 78.246473 secs (137225588 bytes/sec) (131MB/sec)
FreeNAS 0.8 (after tweaks)
[root@freenas] /mnt/storage# dd if=/dev/zero of=./testfile bs=1M count=10K && dd if=./testfile of=/dev/null bs=1M && rm ./testfile
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 62.405507 secs (172058825 bytes/sec) - 164MB/sec write
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 53.998347 secs (198847165 bytes/sec) - 189MB/sec read
And finally, Ubuntu 12.04 with ZFS on Linux:
root@RickNAS:~# dd if=/dev/zero of=/storage/testfile bs=1M count=10K && dd if=/storage/testfile of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 67.544 s, 159 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 31.78 s, 338 MB/s
rick@RickNAS:~$ dd if=/dev/zero of=/storage/testfile bs=1M count=10K; dd if=/dev/zero of=/storage/testfile2 bs=1M count=10K; dd if=/storage/testfile of=/dev/null bs=1M; dd if=/storage/testfile2 of=/dev/null bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 61.1301 s, 176 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 59.9313 s, 179 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 47.6692 s, 225 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 32.2975 s, 332 MB/s
Quite strange, the improvement, but I wasn't going to knock it. So, all looked good, until I tried doing file transfers over CIFS (Windows networking), FTP and SFTP. The results were... abysmal. CIFS and FTP started fast, and rapidly dropped to about 30MB/sec. SFTP was... well, 3MB/sec didn't impress at all. To ensure that ZFS wasn't the issue, I reformatted one of the partitions as ext4, and did another test:
root@RickNAS:/storagetest# dd if=/dev/zero of=./testfile bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 66.944 s, 160 MB/s
root@RickNAS:/storagetest# dd if=./testfile of=/dev/zero bs=1M
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 68.2629 s, 157 MB/s
root@RickNAS:/storagetest#
Not as fast as the raidz was, but still reasonably respectable. Sadly, I got similar results from CIFS etc.
Deciding to stick with ext4 for the moment just to eliminate any extra possible issues, I started to wonder why the performance was so terrible. I ran iperf to my PC, and got pretty respectable speeds out of it, far exceeding the data speeds I was getting in transfer. I went to work trying to improve the Samba speed, to no avail. No matter what I tweaked, nothing seemed to have any effect.
Next step: attempt reinstall natively, after having written out the ISO to USB.
So, reinstalled, after using the PenDrive Linux installer. Got it installed natively this time. Made zero difference. SFTP speeds are pathetic, around the 10MB/sec mark. Across gigabit, that's a bit of a joke, and it's nowhere near maxing CPU, so it's not that as far as I can tell. Did read up online that the driver for the onboard NIC can sometimes be less than perfect and to upgrade to a newer kernel. Did that. Nothing happened. Well, time passed, that's about it. iperf figured look okay though, averaging about 850mbit mark, which isn't too shabby. So there's some sort of bottleneck going on, and it's getting quite frustrating.
So, reinstalled, after using the PenDrive Linux installer. Got it installed natively this time. Made zero difference. SFTP speeds are pathetic, around the 10MB/sec mark. Across gigabit, that's a bit of a joke, and it's nowhere near maxing CPU, so it's not that as far as I can tell. Did read up online that the driver for the onboard NIC can sometimes be less than perfect and to upgrade to a newer kernel. Did that. Nothing happened. Well, time passed, that's about it. iperf figured look okay though, averaging about 850mbit mark, which isn't too shabby. So there's some sort of bottleneck going on, and it's getting quite frustrating.
Next, I've tried to download /dev/zero via SFTP (in FileZilla on the Windows side) and it's settled for now at about 7.2MB/sec download speed. There's something quite clearly wrong. It's using 36% of one CPU for SSHD and 4-5% for sftp-server. It's clearly quite broken. I created a 4GB file in tmpfs in RAM for transfer tests, and vsftpd has slowed to 6.6MB/sec, which is very broken indeed. Make that 6.1MB/sec. Ouch. Rerunning the SFTP transfer resulted in SSHD taking up 96% of CPU at first (fair enough, maxed out) and doing about 40MB/sec, but it rapidly dwindled back to the 8MB/sec mark, using 26% CPU.
Next step: trying local FTP. Hooked in via localhost, retrieved said ram-based file and wrote to disk. 4294967296 bytes received in 26.49 secs (158306.3 kB/s). So, 26s to download the 4GB file, 157.5MB/sec. Seems reasonable. The put was even faster, 13.58 secs for the 4GB. So clearly it's not the software, or at least, not when running locally.
Okay. Back to the basics. Ran a few more longer iperf tests, for 6 minutes at a time each way, speeds were around 820mbit/sec on average. Nothing too spectacular, but nothing too unreasonable, and I was using the PC that they were connecting to at the time as well. Out of curiousity, I installed the CIFS utils and mounted up my Windows network share onto Linux, to copy some files across via the protocol to see what happened. This is where it got a little more confusing.
First, I ran DD to test it, using an mkv video file, 1.6GB. The second time I ran it after dumping the cache, just to make sure, was a little slower than the first time but still fairly respectable.
root@RickNAS:/mnt/win/Rick/Downloads# dd if=./testfile.mkv of=/tmpfs/testfile.mkv bs=1M
1548+1 records in
1548+1 records out
1623222074 bytes (1.6 GB) copied, 16.5486 s, 98.1 MB/s
98MB/sec. Reasonable enough, I think it came out at about 105-110 last time. The same timed operation with cp, after a reboot to flush caches, and a different file of similar size at 1.53GB (as I was struggling to kill caches):
root@RickNAS:/mnt/win/Rick/Downloads# time cp ./testfile.mkv /tmpfs/testfile.mkv
real 1m25.174s
user 0m0.096s
sys 0m6.992s
So, 85 seconds to copy 1.53GB, which equates to just under 18.5MB/sec. Seems a bit harsh. Dropped the cache, and ran it again to see what happened, after the Windows box had cached it:
real 0m17.701s
user 0m0.096s
sys 0m7.248s
C:\temp>dd if=\\.\zero of=.\testfile bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 11.5837 seconds, -0.0 MB/s
Well, that -0.0 MB/s is wrong, but the theory is reasonable. Trying a 10GB file for good measure:
C:\temp>dd if=\\.\zero of=.\testfile bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 812.172 seconds, -246489356887425550000[snip] MB/s
Erm... 0.21MB/sec?! I know I did a bit of web browsing in the meantime... but there's something wrong there. I'm somewhat hoping it's the /dev/zero driver...
Okay. So plan B. Let's try a dd from the Linux box to the Windows box, the other way around.
root@RickNAS:/mnt/win/Rick/Downloads# dd if=/tmpfs/testfile.mkv of=./testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 123.167 s, 13.4 MB/s
... This would be a facepalm sort of moment then.
So what the hell is going on with the Windows box?!
CrystalDiskMark didn't report anything too strange on sequential read/write. 178.8MB/s read, 102.8MB/s write.
After this, I was entirely unable to access any of the mounted files on the Windows side, which left me rather confused. Had to drop this fix into my Windows box, and then all was shiny again. Very odd. So, another try of the DD then...
dd if=/tmpfs/testfile.mkv of=./testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 60.3011 s, 27.4 MB/s
Slightly better, but still a bit rubbish, in all honesty. Again, for good measure:
Resource monitor did show the write speed at 75MB/sec... briefly. Oh.
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 18.7947 s, 88.0 MB/s
... I don't get it. Same again got 87.3MB/s. It shows it as writing. Okay. Let's try another FTP download shall we? Nope, still rubbish.
Okay, time to eliminate the hard disk as an issue. Time to download to a USB HDD and see how that goes.
... Not so well. It chugged along at about 11MB/sec. That's quite poor. But it was fairly consistent, at least. CrystalDiskMark gave it a whole 10MB/sec write too, so no shock. Okay. Throwing a pure RAM disk into Windows to take out the drives as culprits.
dd:
root@RickNAS:/mnt/win# dd if=/tmpfs/testfile.mkv of=./testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 16.2628 s, 102 MB/s
Okay, time to eliminate the hard disk as an issue. Time to download to a USB HDD and see how that goes.
... Not so well. It chugged along at about 11MB/sec. That's quite poor. But it was fairly consistent, at least. CrystalDiskMark gave it a whole 10MB/sec write too, so no shock. Okay. Throwing a pure RAM disk into Windows to take out the drives as culprits.
dd:
root@RickNAS:/mnt/win# dd if=/tmpfs/testfile.mkv of=./testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 16.2628 s, 102 MB/s
Looks far more like normality.
ftp:
File transfer successful, transferred 1,653,435,119 bytes in 16 seconds.
So, that'll be... 98.55MB/sec then. Oh good.
Out of curiosity, I ran sftp, just to see what the performance was like, although I was expecting "dire". I got what I expected, about 20MB/sec. Fair enough, that's CPU-bound. Okay, other way round, just to make sure:
root@RickNAS:/mnt/win# dd if=./testfile.mkv of=/tmpfs/testfile.mkv bs=1M
1576+1 records in
1576+1 records out
1653435119 bytes (1.7 GB) copied, 16.1864 s, 102 MB/s
Looks good. Trying cp commands both ways:
root@RickNAS:/mnt/win# time cp /tmpfs/testfile.mkv ./testfile.mkv
real 0m16.803s
user 0m0.064s
sys 0m7.208s
root@RickNAS:/mnt/win# time cp ./testfile.mkv /tmpfs/testfile.mkv
real 0m15.218s
user 0m0.060s
sys 0m7.432s
So. RAM to RAM is actually fine. I'm going to run Samba to test it out too, to make sure all is as it should be...
... From: 100MB/sec. Copying back 110MB/sec.
Oh.
Okay, so let's try Linux HDD to ramdisk, just to make sure all is shiny as it should be.
... About 110 each way.
So. After all that hunting... all that debugging... all that chasing ghosts, upgrading network drivers and kernels, tearing my hair out... there was nothing wrong with the Linux box at all... it turned out it was my local hard disk. I just hope that this post saves someone else the hours of attempted fixes I tried. It's been a painful journey, but a lesson well learned.
28 April 2012
Displaying Magento tier prices instead of main prices for groups
I've been tackling a particularly annoying problem with Magento, in that I've got specific pricing by groups, imported from another database, but while it'd show the tiered prices in a box, it'd always show RRP on the front page. Fixing it to show the first tier price for quantity 1 on the product page was easy enough (using $this->getTierPrices()), but the tier prices simply aren't present on the grid / listing page, and it really didn't want to let me retrieve them by group either.
So, in case it helps anyone else out at a later date, this is how I got around it, displaying tier prices instead of the main prices on the listing page as well as the main product display page. (Tested on Magento 1.4.1.1, later versions may vary.)
Around line 58, after the following
add:
I also had to alter the following line lower down to avoid displaying "as low as" for items that didn't have lower than the first tier price:
From:
To:
Hope this helps someone out, as it had me scratching my head for a bit, and the online documentation I found wasn't brilliant. In the end the most efficient way seemed to be to rip it straight out of the database. Credit to the stack overflow question that pointed me in the right direction to run the query, modified to select only the customer logged in group and quantity 1 for efficiency.
So, in case it helps anyone else out at a later date, this is how I got around it, displaying tier prices instead of the main prices on the listing page as well as the main product display page. (Tested on Magento 1.4.1.1, later versions may vary.)
Around line 58, after the following
getPrice($_product, $_product->getPrice()) ?>
getPrice($_product, $_product->getPrice(), $_simplePricesTax) ?>
getPrice($_product, $_product->getFinalPrice()) ?>
getPrice($_product, $_product->getFinalPrice(), true) ?>add:
getTierPrices();
if (empty($_tierPrices) && $this->helper('customer')->isLoggedIn()) { // No tier prices found, attempt manual retrieval if customer logged in
$resource = Mage::getSingleton('core/resource');
$query = 'SELECT qty AS price_qty,value AS price FROM '.$resource->getTableName('catalog/product').'_tier_price WHERE entity_id = '.$_id.' AND qty = 1 AND customer_group_id = '.Mage::getSingleton('customer/session')->getCustomerGroupId();
$_tierPrices = $resource->getConnection('core_read')->fetchAll($query);
}
if (!empty($_tierPrices)) {
foreach($_tierPrices AS $tierPrice) {
if ($tierPrice['price_qty'] != 1) continue;
$_price = $tierPrice['price'];
$_finalPrice = $this->helper('tax')->getPrice($_product, $tierPrice['price']);
$_finalPriceInclTax = $this->helper('tax')->getPrice($_product, $tierPrice['price'], true);
break;
}
} ?>
I also had to alter the following line lower down to avoid displaying "as low as" for items that didn't have lower than the first tier price:
From:
getDisplayMinimalPrice() && $_minimalPriceValue && $_minimalPriceValue < $_product->getFinalPrice()): ?>To:
getDisplayMinimalPrice() && $_minimalPriceValue && $_minimalPriceValue < $_finalPrice): ?>Hope this helps someone out, as it had me scratching my head for a bit, and the online documentation I found wasn't brilliant. In the end the most efficient way seemed to be to rip it straight out of the database. Credit to the stack overflow question that pointed me in the right direction to run the query, modified to select only the customer logged in group and quantity 1 for efficiency.
26 March 2012
Exciting adventures with ZeroShell and Hyper-V
Perhaps "exciting" was a little overenthusiastic, but nonetheless I'm happy to publish my rebuild of ZeroShell that runs under Microsoft Hyper-V, at least on our Windows 2008 R2 Core server. This was... an interesting journey, involving a lot of kernel recompilation, patching, initrd editing and banging my head against the desk, but with a bit of shoehorning, the synthetic network adapters and the virtual storage system seem to be working quite well. I've not put it under stress testing or long-term tests yet though.
I've posted up at the ZeroShell forums with the full information on it, and would be quite happy to hear any experience anyone has with it, or if it works for them. I'm hoping to use it to do bonded VPN over our ADSL lines here on a virtualised system rather than an old PC that is currently running pfSense. Porting the firewall rules in is going to be an interesting experience, although I suspect I'll use Firewall Builder on an Ubuntu VM on my PC here, but that's going to be a story for another day.
The direct download link is here. You may need to delete /Database/var/register/system/net/interfaces/ETH* to have your network interfaces appear "normally", as I didn't clean it up before compressing it - sorry about that! More details (and no doubt feedback) will also be on the link above for the ZeroShell forums.
I've posted up at the ZeroShell forums with the full information on it, and would be quite happy to hear any experience anyone has with it, or if it works for them. I'm hoping to use it to do bonded VPN over our ADSL lines here on a virtualised system rather than an old PC that is currently running pfSense. Porting the firewall rules in is going to be an interesting experience, although I suspect I'll use Firewall Builder on an Ubuntu VM on my PC here, but that's going to be a story for another day.
The direct download link is here. You may need to delete /Database/var/register/system/net/interfaces/ETH* to have your network interfaces appear "normally", as I didn't clean it up before compressing it - sorry about that! More details (and no doubt feedback) will also be on the link above for the ZeroShell forums.
Subscribe to:
Posts (Atom)