# On the edge of chaos

Running in the early morning, the dawn sun drives a burnt orange road across the bay. The water’s margin is often the best place to tread, the sand damp and solid, sound underfoot, but unpredictable. The tide was high and at first I thought it had just turned, the damp line a full five yards beyond the edge of the current waves. Some waves pushed higher and I had to swerve and dance to avoid the frothing edge, others lower, wave following wave, but in longer cycles, some higher, some lower.

It was only later I realised the tide was still moving in, the damp line I had seen as the zenith of high tide, had merely been the high point of a cycle and I had run out during a temporary low. Cycles within cycles, the larger cycles predictable and periodic, driven by moon and sun, but the smaller ones, the waves and patterns of waves, driven by wind and distant storms thousands of miles away.

I’m reading Kate Raworth’s Doughnut Economics. She describes the way 20th century economists (and many still) were wedded to simple linear models of closed processes, hence missed the crucial complexities of an interconnected world, and so making the (predictable) crashes far worse.

I was fortunate in that even in school I recall watching the BBC documentary on chaos theory and then attending an outreach lecture at Cardiff University, targeted at children, where the speaker was an expert in Chaos and Catastrophe Theory giving a more mathematical treatment. Ideas of quasi-periodicity, non-linearity, feedback, phase change, tipping points and chaotic behaviour have been part of my understanding of the world since early in my education.

Now-a-days ideas of complexity are more common; Hollywood embraced the idea that the flutter of a butterfly wing could be the final straw that causes a hurricane. This has been helped in no small part by the high-profile of the Santa-Fe Institute and numerous popular science books. However, only recently I was with a number of academics in computing and mathematics, who had not come across ‘criticality’ as a term.

Criticality is about the way many natural phenomena self-organise to be on the edge so that small events have a large impact. The classic example is a pile of sand: initially a whole bucketful tipped on the top will just stay there, but after a point the pile gets to a particular (critical) angle, where even a single grain may cause a minor avalanche.

If we understand the world in terms of stable phenomena, where small changes cause small effects, and things that go out of kilter are brought back by counter effects, it is impossible to make sense of the wild fluctuations of global economics, political swings to extremism, and cataclysmic climate change.

One of the things ignored by some of the most zealous proponents of complexity is that many of the phenomena that we directly observe day-to-day do in fact follow the easier laws of stability and small change. Civilisation develops in parts of the world that are relatively stable and then when we modify the world and design artefacts within it, we engineer things that are understandable and controllable, where simple rules work. There are times when we have to stare chaos in the face, but where possible it is usually best to avoid it.

lovefibre – waves

However, even this is changing. The complexity of economics is due to the large-scale networks within global markets with many feedback loops, some rapid, some delayed. In modern media and more recently the internet and social media, we have amplified this further, and many of the tools of big-data analysis, not least deep neural networks, gain their power precisely because they have stepped out of the world of simple cause and effect and embrace complex and often incomprehensible interconnectivity.

The mathematical and computational analyses of these phenomena are not for the faint hearted. However, the qualitative understanding of the implications of this complexity should be part of the common vocabulary of society, essential to make sense of climate, economics and technology.

In education we often teach the things we can simply describe, that are neat and tidy, explainable, where we don’t have to say “I don’t know”. Let’s make space for piles of sand alongside pendulums in physics, screaming speaker-microphone feedback in maths, and contingency alongside teleological inevitability in historic narrative.

# Paying On Time – universities are failing

Universities are not living up to Government prompt payment targets.  As many suppliers will be local SMEs this threatens the cashflow of businesses that may be teetering on the edge, and the well being of local economies.

I’ve twice in the last couple of months been hit by university finance systems that have a monthly payment run so that if a claim or invoice is not submitted by a certain date, often the first day or two of the month, then it is not paid until the end of the following month, leading to a seven week delay in payment.  This is despite Government guidelines for a normal 30 day payment period and to aim for 80% payment within 5 working days.

I’d like to say these are rare cases, but are sadly typical of university payment and expense systems.  In some cases this is because one is being treated as a casual employee, so falling into payroll systems.  However, often the same systems are clearly being used for commercial payments.  This means that if a supplier misses a monthly deadline they may wait nearly two months for payment … and of course if they are VAT registered may have already had to pay the VAT portion to HMRC before they actual receive the payment.

The idea of monthly cheque runs is a relic of the 1970s when large reels of magnetic tapes had to be mounted on refrigerator-sized machines and special paper had to be loaded into line-printers for cheque runs.  In the 21st century when the vast proportion of payments are electronic, it is an embarrassing and unethical anachronism.

As well as these cliff-edge deadline issues, I’ve seen university finance systems who bounce payments to external suppliers if data is on an out of date form, even if the form was provided in error by a member of university staff.

Even worse are universities finance systems which are organised so that when there is a problem in payment, for example, a temporary glitch in electronic bank payments, instead of retrying the payment, or informing the payee or relevant university contact, the system simply ignores it leaving it in limbo.  I’ve encountered missing payments of this kind up to a year after the original payment date.  If one were cynical one might imagine that they simply hope the supplier will never notice.

The issue of late payments became a major issue a few years ago.  Following the recession, many SMEs were constantly teetering on the edge of bankruptcy, yet larger firms were lax in paying promptly knowing that they were in a position of power (e.g. see “Getting paid on time” issued by the Department for Business, Innovation & Skills, February 2012).

Five years on this is still a problem.  In April last year The Independent estimated that British SMEs were owed 44.6 billion in late or overdue payments(see “The scourge of late payment“).  There is now mandatory reporting of payment processes for larger companies, and recent returns showed that some companies missed prompt payment up to 96% of the time, with bad performers including major names such as Deloitte (see “Ten of the UK’s big businesses that fail to pay suppliers on time get named and shamed by the Government“).

There is also a voluntary “Prompt Payment Code“, but, amongst the signatories, there are only two universities (Huddersfield and Westminster) and three colleges.

Universities are often proud of the way they support local economies and communities: being major employers and often offering advice to local businesses.  However, in respect to prompt payment they are failing those same communities.

So, well done Huddersfield and Westminster, and for the rest of the university system – up your game.

# Rich tea biscuits, sugar lumps and Bournville chocolate

I forgot.

How could I forget?

Memory is a fickle thing, not metal storage shelves, or neat filing drawers, but like the tide throwing up flotsam of your past and then withdrawing, just traces in the sand.

We had been sorting boxes long in storage, and I had made my way through plastic crates full of old screws, hinges, locks without keys, and half window-latches. Some I had collected myself over the years, some I’d inherited from Fiona’s grandpa, and some were my dad’s, accreted through a life as builder, carpenter and maintainer of the old Victorian terrace where I was born. All were coated with that dusty brown patina of age, not the rich iridescent rust of wet, but the dull discolouration that rubs off on your hands and leaves small scatterings on the bottom of tins.

There had been one tin, full of such scatterings, and it had gone into the metal recycling box, amongst others.

I had discarded the brown Tupperware box in which I’d kept my own collection of reusable screws as a boy, a few ‘liberated’ from secondary school desks when it seemed fun to see how many screws you could remove from the lids whilst still leaving them, at least apparently, intact.

What would dad have thought? Maybe some would have been the same desks he had repaired when I had still been in infant school.   A few times a year he would be in our school, repairing desks and chairs – in those days all wood. It is likely he also visited the high school where I eventually ended up, wondering how the lids got loose as a previous generation of school children had a short-lived craze of minor vandalism. How many of the scored and inked images and slogans on the desks where I later sat had been there when he had touched them.

The touch of an object, the feel of it under your fingers, bringing back the past. Only it didn’t, the tin was cast thoughtlessly amongst the decaying ironmongery, detritus of a save-it-just-in-case mentality inherited from those who had seen one or two world wars.

Only after, I remembered.

The tin was long and thin, perhaps ten inches long and two and a half or three wide; square in cross-section; I always assumed it was designed for cream crackers. The lid was large-chequered white and red, with an embossed pattern highlighted in long faded gold, but I only half remember, the way you do with things so intimate, so normal, they are merely the background.

Is it always the way that the things that are closest, most dear, are most easily forgotten?

I took no photograph.

It is gone.

I always say, as a tease, that the smell of meths is the smell of childhood; it reminds me of my dad. And it is true.

He was no drinker, certainly not when I knew him, who knows in his youth. In the sideboard cupboard there was a bottle of cherry brandy that I never recall being opened. Did mum and dad sometimes have a small glass after Jacqui and I had gone to bed? I only ever recall very occasional glasses of sherry at Christmas, and maybe that was only mum.

The smell of meths was surgical spirit; twice a day, regular as the clock that was also wound daily, he would inject insulin. Small bottles with round rubber tops, the needle reused, none of today’s disposable needles, or discreet pens, but his trousers wound down and the needle pressed into his thigh, the skin and needle cleaned with cotton wool soaked in the clear spirit. I wonder how many times he reused the needle; I guess until it was too blunt to break the skin.

When a little older, I recall going together to Cardiff Infirmary, I assume for a check-up – the dull post-war institutional painted corridors, and that smell of hospital … soap and disinfectant, and in those days I’m sure also a touch of meths. I do not know whether it was just once or many times, and why I recall it being just the two of us – maybe it was when Jacqui had started school and I had not, or perhaps Jacqui had gone with mum somewhere, or maybe just that soliloquy of childhood that sees everything through one’s own eyes, forgetting that others were there too.

But in my earliest memories, not the hospital, just the smell, the needle and, in every drawer, handbag, and car shelf, sugar lumps and gold wrapped bundles of Bournville chocolate.

When we went out for the day, or drove away on holiday, mid-morning and mid-afternoon we would always stop for a cup of tea and a bite to eat. Then as now, injected insulin was only half the cure; he had to be careful to eat regularly.

In the summer, on a fine day, we would park the car. Dad would take out a Camping Gaz stove from its blue metal box and the kettle would boil while Jacqui and I played in the sun or sat in the back of the car with a sticky-back-plastic-covered plank as a table.

At other times there were cafés, some with Formica topped tables and counters, others oak panelled – always in those days waitress service. Toasted tea-cakes are still a comfort food.

If we stopped for lunch then we would often have soup with crusty rolls. I’m sure they were in the local bakers too, but I always associate those rolls with days out and restaurants. Jacqui and I would pull out the moist white bread from the middle with our fingers, making mouse houses from the hollow crusts, and then, of course, finish with the crusty parts themselves, still the best portion of any loaf.

Neither dad nor mum took sugar in their tea, but on the table there would always be a bowl full of sugar: sometimes naked white lumps piled high, tempting for a small child, and maybe Jacqui and I would be allowed one each to suck.   Sometimes they came in little paper packets, two bundled together – standard dose for a cup of tea – and, if they did, dad would take a few and add them to his collection, for emergencies, if he felt low on sugar, or if for some reason we were late eating.

Once, I recall dad getting angry and shouting at home, a thing rare enough that I remember it. After a while he and mum realised that he had not eaten, and his temper dissolved as his sugar level rose.

The diabetes was managed, part of the background, one of those things so intimate, so common they are not thought about, but never entirely forgotten.

Once dad broke his toe whilst moving a table in the church schoolroom. His foot was in plaster for weeks, but the worry was always that gangrene would set in.

Only later, after dad had gone, I discovered that one of his brothers had died in the 1920s, still in the early days of insulin treatment when they were trying to understand the correct dosages. The insulin prolonged his brother’s life, but also, in the end, killed him.

I always assume dad’s diabetes was late-onset, otherwise he would never have lived until Jacqui and I were born. Late enough that insulin was better understood. Perhaps it had come at a time of stress, in the 1940s when he divorced his first wife, or when his second wife died.

And the tin?

At home, cups of tea were as much a ritual, dad’s cup bigger than mum’s, but always a cup and saucer; mugs for tea were still many years off. Jacqui and I learnt to drink tea from dad’s saucer. He would pour a little tea on the saucer, blow it and let us sip the cool liquid. It was not just for us, but a trick he sometimes used himself to cool his tea rapidly – a habit from his work as a carpenter to drink quickly in short tea breaks.

With the tea there were no custard creams or bourbons, no chocolate biscuits, nothing iced topped nor anything too sweet, but instead rich tea fingers, thin oval-shaped biscuits with crimped edges. Dad would have two, resting on his saucer and then dunked in the tea until they were soft and warm.

They came in two kinds, one in blue and white packets and slightly lighter in colour, similar in taste to the thicker, round rich-tea biscuits that are more common today; the others in clear packets, with a darker colour and a subtly richer, more savoury, almost nutty taste.

I don’t remember now whether we regularly ate them too as small children or whether they were a grown-up thing. I do recall Jacobs Club biscuits as a treat, always the orange ones. Later as an older child, when I had my own tea, I was always torn between the soft melting texture of dunked finger biscuits, or nibbling them, first around the edge, removing just a few millimetres of the neat crimping, before starting at one end – then, with rodent-like reciprocating teeth, reducing them to sawdust-like powder in my mouth.

The rich tea fingers lived in a tin, and the tin on the sideboard, always.

# Pantoum in Eindhoven

While sorting some old files I came across a small pack of notes, stapled together.  They were clearly written in a bar (the beer glasses are a give away!), and the notes mention Eindhoven.

I then remembered.  On one of my visits to Eindhoven, either teaching USI students at TU/e, or for the Desire conference,  I was sitting with someone at a bar, I think waiting for others to join us and I was describing the pantoum, a Malayan poetry form I had originally read about in “The Making of a Poem: A Norton Anthology of Poetic Form“.  This is a lovely book I was given for birthday or Christmas some years ago, that describes many poetical forms, some common such as the sonnet, others, like the pantoum, that I had never heard of before.

In a pantoum the second and fourth lines of the first stanza become the first and third lines of the second stanza, and then so on for the rest of the poem, like voices calling from verse to verse.  My favourite example was “The Method” by J. D. McClatchy, which stretches the idea of the repeated lines, modifying them slightly to be almost the same, but not quite: perhaps modified, punctuated differently or simply sounding similar but completely different words. For example, the second line of the first stanza is “Seem to pee more often, eat“, which becomes “Sympathy, more often than not” as the first line of the second stanza.

By way of demonstration I tried to write a pantoum on the spot.  There were paper slips on the table, to allow you to write your order to take to the bar, and these became manuscript paper. The lines are very short, which is only fair as I was writing a five line poem on the fly, but I had also clearly forgotten the proper rules (I just rechecked now) as I have six-line stanzas, instead of four-line quatrains.  However, I did manage to get the last stanza to cycle round and use the unrepeated (1st, 3rd and 5th) lines of the first stanza, not bad for a two minute demo 🙂

## The Pantoum of the Guinness in Eindhoven

in the bar
on the street
of old Eindhoven
we sat drinking
Guinness in glasses
dark and deep

on the street
on the pavement
we sat drinking
of the visions
dark and deep
all around us

on the pavement
they told us
of the visions
they saw here
all around us
hidden souls

they told us
as children
they saw here
lonely spirits
hidden souls
are drawn to

as children
in the bar
lonely spirits
of old Eindhoven
are drawn to
Guinness in glasses

# physigrams – modelling the device unplugged

Physigrams get their own micro-site!

See it now at at physicality.org/physigrams

Appropriate physical design can make the difference between an intuitively obvious device and one that is inscrutable.  Physigrams are a way of modelling and analysing the interactive physical characteristics of devices from TV remotes to electric kettles, filling the gap between foam prototypes and code.

Sketches or CAD allow you to model the static physical form of the device, and this can be realised in moulded blue foam, 3D printing or cardboard mock-ups.  Prototypes of the internal digital behaviour can be produced using tools such as Adobe Animate, proto.io or atomic or as hand-coded using standard web-design tools.  The digital behaviour can also be modelled using industry standard techniques such as UML.

Physigrams allow you to model the ‘device unplugged’ – the pure physical interaction potential of the device: the ways you can interact with buttons, dials and knobs, how you can open, slide or twist movable elements.  These physigrams can be attached to models of the digital behaviour to understand how well the physical and digital design compliment one another.

Physigrams were developed some years ago as part of the DEPtH project., a collaboration between product designers at Cardiff School of Art and Design and  computer scientists at Lancaster University. Physigrams have been described in various papers over the years.  However, with TouchIT ,our book on physicality and design (eventually!) reaching completion and due out next year, it felt that physigrams deserved a home of their own on the web.

The physigram micro-site, part of physicality.org includes descriptions of physical interaction properties, a complete key to the physigram notation, and many examples of physigrams in action from light switches, to complete control panels and novel devices.

# Timing matters!

How long is an instant? The answer, of course, is ‘it depends’, but I’ve been finding it fascinating playing on the demo page for AngularJS tooltips. and seeing what feels like ‘instant’ for a tooltip.

The demo allows you to adjust the md-delay property so you can change the delay between hovering over a button and the tooltip appearing, and then instantly see what that feels like.

Try it yourself, set a time and then either move over the button as if you were about to click t, or wondering what it does, or simply pass over it as if you were moving your pointer to another part of the page.

If the delay is too short (e.g. 0), the tooltip flickers as you simply pass over the icon.

If you want it as a backup for when someone forgets the action, then something longer about a second is fine – the aim is to be there only if the user has that moment doubt.

However, I was fascinated by how long the delay needed to be to feel ‘instant’ and yet not appear by accident.

For me about 150 ms is not noticeable as a delay, whereas 200ms I can start to notice – not an annoying delay, but a very slight sense of lack of responsiveness.

# Everything feels easy

Today looked like a good Tiree Ultra day, with 40 mile an hour winds (the odd gust at 50) and occasional shafts of sunshine between driving rain!

So buoyed by knowledge from three weeks ago that I could do it, I took my first run since the ultra.

My left leg is still feeling a little gammy, but with a 40 mph wind at my back I fair sailed along – until I turned round.  Progress on the return leg was … well suffice say I could have walked faster.

I have always avoided running in the rain, but after the ultra I knew I could do it and it wasn’t so bad.  I also had a new rain poof that I’d got for the ultra – good equipment really does help.

There is something liberating about that “it can’t be worse than …” feeling.

When I did the first Tiree Ultramarathon in 2014, it was a year after I’d walked around Wales.  If I got a pain whilst walking there was always the fear that it would be worse the next day, or that it would be the thing that stopped me entirely.

Just over 2/3 of the way round the 2014 ultra I began to get some pain in my right leg.  I’d pulled the Achilles tendon on that ankle a few years before, and so I was a little worried that it would go again.  But I thought, “only 10 miles to go, and it’s just one day. I don’t have to run again tomorrow and the next day; so what if I’m hobbling for a few weeks.

After walking 1000 miles day on day, a single day and mere 35 miles was suddenly less daunting.

Now, knowing I could endure a whole day running with horizontal rain stinging my cheeks, well what of a couple of miles in heavy drizzle and 50 mile an hour winds …

After Tiree Ultra 2017, everything feels easy.

# Tiree Ultra 2017 – what a difference a week makes

Last Sunday I completed my third Tiree Ultamarathon … and definitely the wettest, windiest and boggiest!

However, this Sunday what a difference …

The ultra circuit flows the coast of Tiree taking in almost all of the beaches, but also includes some road sections as well as off-road grass sward and boggy moor. There is relatively little height gain, but Will Wright tries to organise the route to ‘make the most’ of the hills there are.

Previous years have been wonderful weather, light breeze and some sun, enough to be pleasant, but not enough to cause heat problems. However, the fates had been saving their fury, and this year the heavens opened and Odysseus let the western winds loose gathering water from the warm Atlantic and flinging it at us in horizontal sheets.

It was the first time I had every run in the rain so was, well maybe not a baptism of fire, but certainly a dramatic introduction., I had recently bought a waterproof for running in, but it sill had its label on as each wet August day, I thought “well maybe run on a brighter day”. Although everyone says that the right equipment helps, I sort of only half believed it – however, I was amazed at how even driving rain was not a problem.

I only run the Tiree ultra in September and sometimes the Tiree half marathon in May. I always mean to keep on running between, but then I forget, or I am too busy – so many excuses. So, in previous years I haven’t got round to any running until a month before the ultra doubling my distance each week – far from the recommended 10-15% a week increase! To be honest I’ve been very lucky to have not injured myself.

This year I decided to break my habit and be well prepared, so started a whole two months in advance. I wondered if this had been wise as I felt I’d peaked at the end of July and seemed to be going downhill ever since.  However, this year I am definitely hobbling less afterwards and I managed to run every inch of road and beach, with just a few walking sections over bog. This said, when the wind gusted mid to high thirty miles an hour in my face, I would almost certainly have walked faster than I ran.  Indeed on Gott Bay as I ran (very slowly) into the wind another runner was power walking just behind me sheltering in my lee.

One thing I noticed while running was a subtle change in psychology.  After about 10 miles, as pain and exhaustion kicked in, I was aware of myself occasionally wondering if there was any way I could bow out without losing too much face, and then not that many miles later I caught myself thinking “next year I’ll ….” – at that point I knew I was OK!  However, the exhaustion must still have been in my face at mile 17 as the marshal as we came off the beach at Balephetrish said, “you look as if you could do with a hug”.

This year two off-island friends, Albrecht and Alun, also came to Tiree for the Ultra, which was wonderful.  Being a good host I of course let them finish ahead of me by an hour or so 😉

Albrecht has already booked for next year, but not sure if Alun is convinced!

However, the weather can only be better.

# running on the verge

Tiree Fitness Facebook – Photo by Alan Millar

In a week and half’s time I’ll be joining about 250 others on the Tiree Ultramarathon, running around the edge of Tiree, which is itself on the Atlantic edge of Scotland.   Some of this will be on beach and moor, but some along single track roads, where you often have to step onto the grassy verge as cars go by.

Running on the verge has its own challenges which I’m sure are shared by many rural areas as well as Tiree.  For those coming to the Tiree Ultra or running (or cycling) in rural areas, here’s my short guide to the hazards of the verge.

on narrow roads do stop – Some roads do have space for a car to pass a runner or cyclist, but it can be close especially if you are a little tired and ‘wandering’ a little as you run.  So usually best to stop … and you get a moments breather 😉

beware the ragged tarmac edge – It is tempting to just squeeze to the left and keep going, but the tarmac often peters out, this is worst of you are cycling as the wheel can slip off the road and get trapped in the furrow between tarmac ad grass (cyclists have ended up in hospital!), but you can also trip when running … and you don’t want to fall into the path of the car that is passing.

Tiree Fitness Facebook photos – verges are not the only road hazard

running on the verge – I know many will ignore this, but just don’t.  They seem wide, tamer than running on full off-road terrain, and well within the capabilities of a off-road bike.  However there are often drainage channels hidden by the long grass – these can be a foot or more deep and can be invisible.  Even when there isn’t a deep drainage channel running parallel to the road, there are often smaller drainage channels running outwards from the road; these are typically only a few inches deep, but just designed to trip you up.  The one possible exception is where someone has mown the verge outside their house, but even then be careful of the cross-channels as they often aren’t obvious even on mown grass.

stepping onto the verge – At the risk of sounding like your granny, still take care!  I have stepped off the road and, even looking down at the ground as I did so, my foot has disappeared into a channel and I’ve almost sprained my ankle … and that was standing still not running.  On the bike be even more careful, you stop, put your outer foot into what you believe to be grass and … on a bike there is little you can do apart from topple full head over heels … and, yes, I know because I have done it.

standing on the verge – Will it never stop!  Yep, even standing has it’s dangers.  On Tiree it is normal to wave to those passing, friend and stranger alike.  However, if you are a little tired twisting round can put you off balance.  Don’t feel embarrassed to put a hand on a fence post to keep you sure footed, better than stumbling back into the path of that nicely waving driver.

stepping off the verge – Do take a peek back down the road before stepping back onto the tarmac.  Tiree is windy and when the wind is coming from in front it is hard to hear cars from behind, as a car passes you it is easy to just step back, but often there is a second car driving in convoy, especially when the road has had a lot of obstacles (such as runners), so that cars catch up with one another.

… and then if you survive the verges

… there is just Dun Mor to climb …

# Solr Rocks!

After struggling with large FULLTEXT indexes in MySQL, Solr comes to the rescue, 16 million records ingested in 20 minutes – wow!

One small Gotcha was the security classes, which have obviously moved since the documentation was written (see fix at end of the post).

For web apps I live off MySQL, albeit now-a-days often wrapped with my own NoSQLite libraries to do Mongo-style databases over the LAMP stack. I’d also recently had a successful experience using MySQL FULLTEXT indices with a smaller database (10s of thousands of records) for the HCI Book search.  So when I wanted to index 16 million the book titles with their author names from OpenLibrary I thought I might as well have a go.

For some MySQL table types, the normal recommendation used to be to insert records without an index and add the index later.  However, in the past I have had a very bad experience with this approach as there doesn’t appear to be a way to tell MySQL to go easy with this process – I recall the disk being absolutely thrashed and Fiona having to restart the web server 🙁

Happily,   reports that for MyISAM tables incremental inserts with an index are no worse than bulk insert followed by adding the index.  So I went ahead and set off a script adding batches of a 10,000 records at a time, with small gaps ‘just in case’.  The just in case was definitely the case and 16 hours later I’d barely managed a million records and MySQL was getting slower and slower.

I cut my losses, tried an upload without the FULLTEXT index and 20 minutes later, that was fine … but no way could I dare doing that ‘CREATE FULLTEXT’!

In my heart I knew that lucene/Solr was the right way to go.  These are designed for search engine performance, but I dreaded the pain of trying to install and come up to speed with yet a different system that might not end up any better in the end.

However, I bit the bullet, and my dread was utterly unfounded.  Fiona got the right version of Java running and then within half an hour of downloading Solr I had it up and running with one of the examples.  I then tried experimental ingests with small chunks of the data: 1000 records, 10,000 records, 100,000 records, a million records … Solr lapped it up, utterly painless.  The only fix I needed was because my tab-separated records had quote characters that needed mangling.

So,  a quick split into million record chunks (I couldn’t bring myself to do a single multi-gigabyte POST …but maybe that would have been OK!), set the ingest going and 20 minutes later – hey presto 16 million full text indexed records 🙂  I then realised I’d forgotten to give fieldnames, so the ingest had taken the first record values as a header line.  No problems, just clear the database and re-ingest … at 20 minutes for the whole thing, who cares!

As noted there was one slight gotcha.  In the Securing Solr section of the Solr Reference guide, it explains how to set up the security.json file.  This kept failing until I realised it was failing to find the classes solr.BasicAuthPlugin and solr.RuleBasedAuthorizationPlugin (solr.log is your friend!).  After a bit of listing of contents of jars, I found tat these are now in org.apache.solr.security.  I also found that the JSON parser struggled a little with indents … I think maybe tab characters, but after explicitly selecting and then re-typing spaces yay! – I have a fully secured Solr instance with 16 million book titles – wow 🙂

This is my final security.json file (actual credentials obscured of course!

```{
"authentication":{
"blockUnknown": true,
"class":"org.apache.solr.security.BasicAuthPlugin",
"credentials":{
"tom":"blabbityblabbityblabbityblabbityblabbityblo= blabbityblabbityblabbityblabbityblabbityblo=",
"dick":"blabbityblabbityblabbityblabbityblabbityblo= blabbityblabbityblabbityblabbityblabbityblo=",
"harry":"blabbityblabbityblabbityblabbityblabbityblo= blabbityblabbityblabbityblabbityblabbityblo="},
},

"authorization":{"class":"org.apache.solr.security.RuleBasedAuthorizationPlugin"}
}
```