Ludicrous speed

April 7, 2017 Leave a comment

Just a quick post for some storage bragging rights πŸ™‚

I’ve upgraded my home PC with a pair of SanDisk FusionIO ioScale 410GB PCIe flash adapters. And I thought SSD was fast πŸ™‚


Here’s a quick benchmark using the Disk utility in Ubuntu 16.04.


For comparison, here the same but for a SanDisk 256GB SSD.


And a comparison to an old 1TB SATA drive that I’ve still got running.


Finally, here a benchmark on a software RAID-5 array, running on three old 300GB SATA drives.


The only downside is that the drivers for the FusionIO cards aren’t in the mainline Linux kernel, so I need to recompile them whenever I upgrade the kernel version.

Categories: Uncategorized

Dock in the Box

April 12, 2016 Leave a comment


I finally got around to re-imaging my Raspberry Pi cluster to run Docker. I’ve used the Hypriot image, which worked flawlessly. And retrofitted some scripts to manage soft power control for the cluster nodes via the ATXRaspi board.

# docker-machine ls
dspi0 generic Running tcp://x.y.z.0:2376 dspi0 (master)
dspi1 generic Running tcp://x.y.z.1:2376 dspi0
dspi2 generic Running tcp://x.y.x.2:2376 dspi0
dspi3 generic Running tcp://x.y.x.3:2376 dspi0
dspi4 generic Running tcp://x.y.z.4:2376 dspi0
dspi5 generic Running tcp://x.y.z.5:2376 dspi0
dspi6 generic Running tcp://x.y.z.6:2376 dspi0

Next I need to figure out what this Cetacean cluster can do, and how to do it…

Kernel Version: 4.1.17-hypriotos-v7+
Operating System: linux
Architecture: arm
CPUs: 28
Total Memory: 6.984 GiB

I do want to get Ceph running on the cluster again, but will probably run that directly on the base Linux OS rather than in Docker. It can then present storage to the various Docker containers.

Finally, for no particular reason, I cut up a heatsink from an old video card, and stuck them onto the Pi nodes.

Categories: Docker, Raspberry Pi

Napkin notes

April 5, 2016 Leave a comment

Find disk capacity used by files older than a given date:

find . -type f -atime +30 -print0 2>/dev/null| du -hc --files0-from - | \
tail -n 1

And wrap it in a loop to show capacity by various dates:

for a in 30 90 180 365 730 1460 11680; do \
echo -n "Files older than $a days = "; \
find . -type f -atime +$a -print0 2>/dev/null| \
du -hc --files0-from - | tail -n 1; done


Categories: Uncategorized

Back to school for OpenStack training

July 17, 2015 Leave a comment

This week I’ve been fortunate enough to attend a four day Red Hat OpenStack Implementation course.

I’d recommend this course to anyone interested in developing their skills and familiarity with Red Hat’s OpenStack distribution. I found the course delivery and materials to be excellent, and labs, lots and lots of lab exercises. Manually installing and configuring all the common services; Keystone, Glance, Nova, Neutron, Cinder, Heat, Swift, Ceilometer, Horizon, as well as Gluster, RabbitMQ, and MySQL.

Below is a gratuitous screen shot of the completed labs – a small victory that was made hollow by then seeing PackStack automate a similar deployment in about 30 minutes.

RH-OS course

Now to get it all deployed on my Pi cluster at home πŸ™‚

Categories: OpenStack

A bucket of squid

June 10, 2015 5 comments

Since my last post I’ve been tinkering with my little Pi cluster a bit, but mostly trying to sort out a better case and power supply. Although I do like the Lego aesthetic it’s far too likely to be randomly disassembled by my kids! The power supply was also an issue, as the little USB powered hub only had a 3A supply and was struggling with six Pis.

So I’ve rehoused the cluster into the case shown below, and I think now have a (still WIP) good solution for the power supply. The picture shows five Pi 2 B cluster nodes (plus one Pi B+ for cluster control), though I’m planning to add a sixth Pi 2 B cluster node on the left hand side. It’s hard to see, but there’s also an eight port Ethernet switch under the Pis.

## See the bottom of this post for updates ##

Pi chassisUnfortunately the case was damaged in transit, and is now held together with super glue! To their credit the vendor quickly provided a full refund, and I’m ordering a replacement. Until that arrives I’m using the damaged case as a proof of concept.

Pi casePi caseApart from the damage in transit, I’m very happy with the case. There was a 120mm fan under the top vent, but I removed that and the system remains cool-ish with only passive air flow.

For the power supply I’m using four quad-USB car charging adapters, all connected to a laptop power supply. You can see them in the top left and right of the case photos. The USB charging adapters can tolerate between 8-20V and output 5V and 6A, 30W, across four USB ports each. Perfect for my small Pi cluster, and with the potential for an in-car mobile Pi cluster (just kidding!).

USB powerCurrently the power supply is still a bit primitive, with no power switch or soft power down option. So I need to manually shut down all the cluster nodes before disconnecting the power. Which brings me to the WIP section…

I’m planning to use an ATXRaspi board to manage power to the Pi control node, and modify the startup/shutdown process to then manage power to the six Pi cluster nodes. The control node will connect via GPIO to a solid state relay to switch the cluster power on and off. I’m considering using two SSRs so that I can selectively fail half of the cluster nodes.

Pi case power planPower On

  1. Connect mains power, no power to RPi nodes (option to continuously power Ethernet switch)
  2. Press ATXRaspi switch, enable power to RPi control node (and Ethernet switch) and trigger startup script
    1. Startup script activates the DC relay and enables power to the RPi cluster nodes

Power Off

  1. Press ATXRaspi switch, this triggers shutdown script on RPi control node
    1. Shutdown script executes remote shutdown of RPi Cluster nodes via SSH
    2. Shutdown script deactivates the DC relay and powers off the RPi cluster nodes
    3. Shutdown script completes the shutdown and powers off of the RPi control node
      (and Ethernet switch)
  2. Disconnect mains power

I’ll update next when the parts arrive and I see if this actually works… πŸ™‚

## Update 1 – It Lives! ##

I found some time this evening to pull the cluster apart and test the ATXRaspi and relays that arrived last week.

Looks good! Button control works great and the control node startup (/etc/rc.local) triggers the relays via the GPIO connections.

Testing ATXRaspiNow to put it all back together and test the cluster node shutdown scripts.

## Update 2 – Back in the box ##

Here are some photo’s of the reassembled cluster, now with working soft power on/off controls.

Pi cluster

Front top view – showing power button and ATXRaspi (lower right side)

Pi cluster

Front left view – showing the USB power adapters that are controlled by the relays

Pi Cluster

Front right view – showing the power control relays

Ceph is running nicely, and I’ve updated the CRUSH map to split the pg copies between the odd/even nodes, as they hang off different USB power supplies.

ceph_osd_treeAnd some glamour shots πŸ™‚

Pi ClusterPi ClusterPi ClusterI’ve also uploaded the pictures here –

Categories: ceph, Raspberry Pi Tags:

My little Cephalopony

April 9, 2015 Leave a comment

In order to teach myself a bit about Ceph, I’d like to introduce my modestly scalable, low performance, gigabyte-scale Raspberry Pi storage cluster!

Ceph Pi clusterThat’s five Raspberry Pi v2, each with the standard quad-core ARM Cortex-A7 CPU (over-clocked to 1 GHz), 1 GB memory, and four 8 GB USB memory sticks for storage. When I was a lad that was a lot…

Installation was embarrassingly easy, mostly following instructions from here, though I used Minibian as the OS instead of Raspbian. Along the way I also discovered the Mosh shell, which makes remote connection over varied, slow, and intermittent networks tolerable.

Obviously this is an educational build and not intended for serious use, though it’s already clear that 8 GB USB sticks are too small. Their appeal was low cost and the “Christmas tree” effect of so many blinkenlights.

ceph@piceph1:~$ ceph -s
cluster 661a43a5-e9cc-4350-851d-109d07c44773
health HEALTH_OK
monmap e1: 1 mons at {}, election epoch 2, quorum 0 piceph1
osdmap e24: 5 osds: 5 up, 5 in
pgmap v49: 192 pgs, 3 pools, 0 bytes data, 0 objects
25694 MB used, 9240 MB / 36923 MB avail
192 active+clean

I’ll update with more detail as I figure out just what to do with this “data centre in a lunch box” πŸ™‚

Categories: ceph, Raspberry Pi

Custard On Top

September 24, 2013 Leave a comment

I’ve started a new IBM Redbook residency this week. A bit different to those that I’ve previously worked on, as work and family commitments don’t allow for travel. So I’ll be working remotely, and part-time, from Australia while the rest of the team are in Mainz, Germany.

The aim is to produce a new Redbook on IBM N Series Clustered Data ONTAP (cDOT), and to update the existing N Series hardware and software guides. I was also on the team that produced the previous update to the N Series hardware and software guides.

I’d describe myself as very experienced with Data ONTAP 7 (or DOT 8 in 7-mode). I was first NCDA certified in 2005, and have since collected an embarrassing array of NetApp certs. I also worked at NetApp as a technical instructor, and wrote the original NCDA certification study guide. But so far my experience with Clustered mode has been minimal.

For me, this is a great opportunity to keep my technical skills current, whilst contributing my experience and new knowledge back to the book.

I’ll update this post over the next few weeks to document my experiences.

Week 1 – Back to school

The first week was spent in a Clustered ONTAP 8.2 Administration course, to get us all up to speed with the new features. The course was presented via WebEx in a “virtual live” format. The WebEx format worked well, with a very knowledgeable instructor, and plenty of opportunity to ask questions.

The gotcha for me, is that the course was scheduled for Central European time, since that’s where the rest of the residents are located. That’s 5pm-1am here in Melbourne, Australia. With my day job, and family too, this made for a _very_ long week.

My first impressions of cDOT are very positive, with much of the traditional ONTAP goodness still familiar, though enhanced in many ways. At the same time, the change from Active/Active ‘Clustered’ controllers, to a scalable cluster that is built from ‘HA Pairs’ has altered the behaviour of many features.

I felt a bit like a wide-eyed Dorothy in the Wizard of Oz – we’re not in Kansas anymore!

Toto, I've got a feeling we're not in Kansas anymore

Here’s a quick list of some changes in Cluster mode:

  • Clustered ‘HA Pairs’ replace the traditional dual Active/Active controller design.
  • Fail-over within a HA Pair is improved by using Storage Fail Over (SFO) instead of Cluster Fail Over (CFO).
  • Different resource types, such as aggregates, volumes, and network ports & interfaces,Β  are now bound either to physical nodes, vservers, or are cluster-wide.
  • Vservers, otherwise known as Storage Virtual Machines (SVM), replace the vfiler Multistore feature.
  • Whereas a 7-mode Virtual Interface (VIF) is bound to a physical node, its replacement, a Logical Interface (LIF), is bound to a logical vserver (cluster-wide).
  • SnapMirror loses its Qtree and Synchronous modes, but gain several other features, such as LoadSharing mirrors
  • SnapVault is now based on Volume SnapMirror, so retains the storage efficiency features of the source (deduplication and compression)
  • Some features from 7-mode are not available, but may be expected in future versions. (e.g. SnapLock, MetroCluster, SyncMirror, etc).

There are many more that I’ve not mentioned in the above list, but for those you’ll need to read the book (or at least the updates to this post πŸ™‚

Week 2 – Meet the team

It’s only the start of the week, but here’s a screen shot of the kick-off meeting. We used Google Hangouts as a good way for the team to ‘meet’.

My fellow residents are (from left to right): Christian Fey, Danny Yang, Michal Klimes, Roland Tretau (leader, main screen), myself, and Tom Provost.

Roland TretauResidents

This meeting was a brain storming session to create a draft Table of Contents for the book, main parts, chapter topics, etc. Once the ToC is agreed we’ll assign different chapters to each team member to complete.

I’m going toΒ start on the updates to the existing H/W guide and 7-mode S/W guide. This is a good fit for working remotely as it’s separate from the Cluster mode book, though there will be common sections that should be easy to adapt. Once I’ve finished updating the existing books I’ll rejoin the rest of the team.

All Redbooks are developed in Adobe FrameMaker, so one of the first tasks for any resident is to install the FM software and the ITSO custom toolkit. The ITSO have developed an applet called QAM (Quick Access Menu) that automates many book creation tasks. This also ensures that the authors, who are often new to FrameMaker, always follow the ITSO style.

Mock up of the cDOT cover

By the end of the week I’ve merged the contents of both the h/w and s/w guides into the new book files, and have started reviewing new material for inclusion.

The main challenge so far hasn’t been working remotely, but in trying to find time to work on the book part-time. I’m still doing my day job too, and anyone who’s worked at IBM knows that already requires 100% effort!

Week 3


Week 4


Week 5


Week 6


Categories: Storage, Training

Bus timetables and storage efficiency

March 28, 2011 1 comment

Sometimes storage insight comes from the most unlikely of places…

I recently attended an IBM IT Architect networking event where the topic was IBM’s Smarter Planet initiative. One of the talks, by IBM software Architect Andy Heys, was on the topic of Smarter Transportation. His talk included an example of IBM’s Traffic Planning Tool and how IBM had developed a traffic prediction model for the Singapore Land Transport Authority (LTA).

This gave me an unexpected insight into the parallels between Singapore bus timetables and storage thin provisioning. Seriously!

The bus bit

The story goes, that after having developed a system to accurately predict traffic flow for up to 60 minutes into the future, IBM then suggested taking the next step, which was to start adjusting traffic signals so as to reduce congestion. It’s a natural assumption that studying and predicting traffic flow would then be used to improve that same traffic flow…


At this point, says Andy, “we realised that they had more insight into the problem than ourselves”. The LTA was more interested in improving bus arrival time prediction than in directly improving traffic flow. That’s because they understood that any improvement in traffic flow would simply attract more traffic to the roads, which would rapidly return them to the same congested state, and probably worse.

Their plan was to attract more people onto public transport by reducing the schedule variance and unpredictability of bus arrival times. After all, who wants to wait at a bus stop if they are unsure of when the next bus will arrive. More people taking the bus would also reduce the number of cars on the road, and indirectly improve traffic flow too!

The storage bit

At first glance, traffic prediction has some obvious similarities to storage capacity planning – measuring utilisation and plotting trends. But I think the more interesting parallel is the competition between high storage utilisation (ie. road congestion) and storage efficiency tools such as thin provisioning, compression and deduplication (ie. methods for improving traffic flow).

Just like with traffic planning, storage also suffers from unexpected side effects, such as how increased available capacity (eg. via thin provisioning, or deduplication) is rapidly consumed by newly attracted workloads. With the same end result being a larger and more difficult problem to manage.

So, what is the analogue for public transport in the storage world? How do we improve capacity management, not by increasing storage density, but by reducing the storage burden?

It’s at this point that the conversation changes to be about data archiving, which serves the dual purpose of managing capacity and “getting cars off the road”.

Of course, I’m not saying that storage efficiency tools are unimportant, just that they are not enough on their own. They provide great value by decreasing (sometimes dramatically) the amount of physical storage that is required.

Back to the buses

The difficult question then is “how to encourage good behaviour (public transport / archiving) when the demand is always for more cars (increased storage capacity, greater performance, lower cost, and faster provisioning)”?

To use another Singaporean traffic analogy, this is already a solved problem, where congestion charging is used to toll traffic within the CBD area (ie. Tier-1 storage).This may never be a popular tool, but it is certainly an effective one. Tolls (and archiving) are like broccoli – good for other people πŸ™‚

I suppose that I should highlight that I’m not really advocating for charge-back, but rather for intelligent capacity management.

While it might be difficult to introduce chargeback for an existing storage environment, might it be possible to introduce new capacity, capabilities, and costings which encourage proactive storage management? If it’s easier, cheaper, and faster to catch the bus rather than to drive a car, then more people will choose the bus willingly.

The (unfortunately very dated) diagram below shows the impact of congestion charging on Singapore CBD traffic over a 20 year period. I like how similar this looks to the archetypal chart for storage growth (always increasing), except the doubling period is about ~18 years, not ~18 months as for storage (YMMV).

Effects of ALSData is scaled to 100% for base year 1975
Blue: Car Population, Yellow: AM Inbound, Red: PM Inbound

Can a similar model of selective charge-back (or relative “charge-less” for lower storage tiers) work in enterprise storage? It always amazes me that it’s easier to impose tolls on public roads than on commercially managed storage. For many environments which don’t practice storage charge-back, perhaps the answer is simply to make the “public transport” option more appealing?

PS/ A sub-title for this post might be “Is public transport (ie. archiving and/or tape) dead – no way!” πŸ™‚

Categories: Storage

First post!

February 3, 2011 Leave a comment

Hello, and welcome to this, the launch of my shiny new storage blog!

Columbia launch

I’m planning to discuss anything and everything storage related; from industry trends to technical nuts and bolts. I’ll probably also cover some training delivery and instructional design, as these are also professional interests of mine.

Note that although I’m currently employed with a storage vendor (IBM), this is a personal blog of my own thoughts, actions, and opinions. According to IBM’s social computing guidelines, I should say something like: “The postings on this site are my own and don’t necessarily represent IBM’s positions, strategies or opinions.”

Steven P.

PS/ If you want to know a bit more about myself you should look me up on LinkedIn.

Categories: Uncategorized