The data-driven organisation - an allegorical tale...

BAD PRACTICE: THE DATA DRIVEN ORGANISATION
An economic development agency can’t understand why drop-out rates for its training programmes for unemployed young people are so high. So they commission a survey to find out, spending £100,000 on consultants to survey training programme participants and drop outs, conduct focus groups and research with a final report.

BUT!

A little bit of intelligence work on the local and national economy suggests that at the present time, more and more young people are staying in formal education, and the labour market offers fairly good job opportunities and prospects for young people, whether unemployed or just leaving school. Anecdotal feedback from training providers tells the agency that the entrants into the training programmes are people who cannot continue in education or cannot access jobs – they are the least qualified, and least employable young people. The training programmes were originally designed to cope with the average unemployed young person 20 years ago, and not aimed at those who have more acute problems in attaining qualifications and accessing work. Therefore the training programme does not meet the needs of the client group today, as it did 20 years ago. The issue is that the labour market has changed, while the training programme has stayed the same. Spending a few days finding this out is cheaper and provides more insight than commissioning a £100,000 research project.

THE MORAL OF THE STORY

The moral is that gathering intelligence on its own doesn’t provide you with answers. You must go one step further and ask what intelligence is telling you about your activities, the market and environment that you operate in. Initially, its best to analyse information already at hand to see if it answers your questions than to go and collect more and more data.

THE ANALYSTS EXPERIENCE – COMMON ISSUES AND PROBLEMS WITH SURVEYS
This analyst gets angry!

I have lost count the number of times that I, or a colleague, have been administering and delivering a survey for a range of economic development organisations and they have asked for a bigger sample for their local area – AND – they have not been able to justify or give any rationale for the survey boost! Say the survey boost costs £100,000 and there is no reason for it – what’s the point? Then the local agencies tell us that its not accurate enough for local purposes - we ask ‘for what local purposes?’ – too often they couldn’t come up with any answer.

But – before we get too bolshy – people often have great difficulty expressing their needs and why they want things. A good analyst gets to the bottom of this need, if there is one. A good analyst asks the questions ‘what do you want the extra sample boost for?’ – as it tells them where the client or partner is coming from. And then the answer may not lie in a sample boost at all.

parchutists in the civil service

Ah um, so a certain government department whom used to do a lot in a certain area of economic development and business stuff devolved a load of it down to the regions. And now they have appointed regional officers. Besides the point that well, why do we need a regional officer if its all devolved now, the officer doesnae have much of a scooby doo.

I have heard the new regional officer speak, and well, they haven't really mastered their brief yet! It was amongst experts, and was a cringeing moment. Mind you, it will be easy to fob them off as they won't be able to tell a red herring until someone else slaps them in the face with it.

What is it about the UK civil service where they radically change jobs as often as Portsmouth FC gets a new manager? Its annoying. Like we have people who come and work in employment policy who previously worked in policy for liquor export licensing or something as bonkers as that.

Where I worked previously, in Scotland, the head of skills and employment changed about three times in three years. Can you believe it. They'd have to be briefed every time. I remember a presentation by one, and thinking my goodness those ideas are at least 5 years old! they obviously had an early edition of the textbook!

What is it about our Civil Service where people are encourages to be generalists and there are few senior specialists?

how to do evidence-based policy - some pointers

GOOD PRACTICE: PROVIDING THE EVIDENCE FOR ‘EVIDENCE-BASED POLICY’

I was extremely privileged to work for several years in an applied research unit which had been tasked with using evidence to influence decision making. This applied to the decision making of a varied bunch – from policy makers and their executives, to careers advisers and their clients. The idea was to collate and analyse labour market evidence and to tell people really what it meant – i.e. to improve their knowledge of the labour market to inform their practice or their actions and help them make better informed decisions.

EVIDENCE-BASED POLICY IN ACTION

It was extremely challenging to start to make evidence more useful. Traditionally, economists and analysts in the area in which we were working had typically delivered long-winded reports with innumerable tables and complicated graphs, and had shied away from analysing the key insights and messages from evidence. It was left to the practitioners to try and become analysts themselves – a flawed assumption. Of course, there were a few practitioners with the time and energy, that could do the analysis, but 99% couldn’t and it was unreasonable to expect this.

It was clear we would have to take a different approach. We did this by:

* Getting to know the market for analysis, intelligence and information – just what did people want from economic analysis and in what format or style? They told us they didn’t use voluminous reports – they sat on the shelf. We actually segmented the market in terms of sophistication of understanding and use of intelligence. Quite quickly there was one market we weren’t interested in catering for – the experts. The experts could do their own analysis, and were comfortable with long winded reports but represented a very small user base for our services who mostly had limited influence on policy or delivery. There was no payoff in catering to this user base, apart from being well thought of by our peers. I think it is a mistake of many research units in the public sector that they try to emulate experts as they think it’s the best way to go about things – but they ignore the bulk of their consumer base which is the middle ranking public sector official or service delivery agent.

*Getting to know how people could use intelligence to better inform their decisions – what do they need to know and when? This involved identifying a few key constituencies and opening up lines of communication about their needs – for example, we did this with the Further Education College sector and with the Careers Services sector. This eventually led to collaborative products that were market tested with clients before public release.

* People wanted short reports! They were fed up with 100-page reports – they wanted much shorter reports – in the case of careers advisers, they wanted one or one-half page news stories! We started to provide a hierarchy of reports – we would have the 100 page reports and tables that were necessary in the first place on which to do the analysis – we provided these but didn’t make a song and dance about them; we then provided a 20-30 page easy to read report; then a 5-10 page executive summary; and then a few 1-page tabloid articles covering the main points of analysis, findings and messages in an engaging way.

* People wanted key messages and insights directly relevant to them, but to be also assured of the robustness and quality of the research behind these messages and insights. Ok, we needed to provide short and accessible reports, but the work behind them better be robust and comprehensive. We achieved this through setting high quality standards for ourselves and our contractors. Contractors whose quality of work was inadequate were not rehired. We also engaged in some training for presentations with media people – this meant that we made good external presentations and that we could stand up and be credible as well as informative. We tried to establish and maintain a quality standard for our ‘brand’.

* People wanted ad hoc or regular access to key statistics that were up to date. They wanted this for their reports, their boss’s presentations, or their funding applications. They also wanted all the background information about the statistics – what they meant, the source, the accuracy. Plus, there was a selfish motive by us, the staff – for who wants to be bombarded with requests for the latest unemployment rate or other figures? We had some online resources developed which would provide a range of key indicators in an easy to use internet based toolkit.

*We wanted people to use evidence – as such we had to ensure it was relevant, they would read it, and would find it beneficial.

THE MORAL OF THE STORY

People want information, intelligence and economic analysis for a reason or a purpose. To meet this it must be ‘fit for purpose’ – and that doesn’t just mean technically correct or accurate (we’re taking this as a given, folks) but in a format and style which genuinely makes a decision or service delivery easier to make or enhanced in some way.

An academic gave our little applied research department a favourable review in a letter to a local newspaper – he said ‘evidence based policy needs evidence’ and congratulated us on sourcing and providing the evidence. I would like to take this further by saying ‘evidence based policy needs accessible, meaningful and relevant evidence’. It is no use providing 200 pages of evidence if it is irrelevant or if no-one is inclined to read it.

For some analysts, they may feel ‘cheapened’ or that they are selling themselves too far to provide their analytical insights in a one page tabloid article. For me, this is unnecessary snobbishness. I would rather do a piece of research and have 1,000 people read it and benefit from it than just 10 fellow analyst who probably knew it all anyway.

USEFUL LINKS:

The best Careers labour market intelligence resource I have ever seen: Oregon Labour Market Information Service - www.qualityinfo.org/olmisj/OlmisZine

Using evidence to inform policy in the labour market in Scotland: www.futureskillsscotland.org.uk

Interesting research - survey of 4000 London businesses

A decent survey with a decent sample (report here: http://www.lda.gov.uk/server/show/nav.001005005002)

Some interesting findings about business growth - turnover, profitability, jobs. Where they did some nice multivariate analysis.

Key messages:

• The London Annual Business Survey 2004 revealed that growth is more likely to occur in businesses that have one or more of the following characteristics of behaviours:
– employ at least 5 employees, are sole proprietorships or partnerships, or private or public limited companies;
– use business planning tools and formal management information systems;
– are expanding their markets in the UK or globally;
– have successfully gained external finance, and/or are investing in equipment;
– collaborate with other organisations, mostly other businesses;
– undertake R&D and innovation activities;
– adopt a range of competitiveness strategies such as adopting IT, launching of new products and services, expanding markets, and/or cutting production costs; and/or
– use Business Link and other public sources of business advice or information.

• High performing businesses perform well across most measures of performance – they tend to perform well in turnover growth, profitability growth and productivity growth at the same time.

• Businesses less than 10 years old; in financial services, business services or ICT activities; or with a mixed ethnic ownership were more likely than average to have experienced turnover growth.

• IT investment has high positive impacts, and many businesses are actively engaged in this. Most businesses use IT regularly.

• The most commonly cited methods for enhancing competitiveness were: to improve customer/client relations; advertising/marketing strategy; adopting IT; and launching new products and services.

Nice fluffy things in business support

Wouldn't it be NICE to have Business support services and systems that…

- Have leadership, clarity of role, priorities

- Are tailored to the locality/region's situation

- Offer a better customer experience

- Deliver quality advice and information

- Work with the grain of the market

- Use the advantages of locality/region as a business location

- Achieve better impacts to the economy for our money

- Deliver to priority groups and growth businesses

Ah as Gilbert from Gilbert's Fridge used to say "you can aalways dream..."

Mondays

Ah some Monday's one can't be arsed can one? A day of watching the clock. I like my job, but sometimes going on holiday permanently must appeal to everyone. Especially on a dark November morning after a weekend of unremitting rain.

My vernacular is being influenced by the queen since I took a trip up an East London tower block to view the Olympic Site. Following in the footsteps of HRH a few weeks ago.

Anyhow - thinking about innovation these days. What's it all about in RDAs? not many seem to have got their act together really. Its all a bit wooly - promoting innovation, erm well its really promoting productivity and use of new products and processes. Innovation is all a bit too vague really. In Scotland it used to mean spending large dollops of cash on science parks and technology institutes. The RDAs seem to have all their development/capital cash tied up in regeneration stuff.

And what about regeneration - the SRB is drying up next year! I am a bit sceptical that regeneration really achieved much for the money spent in lots of areas. Some good projects, but also a huge rush to fund anything to get the cash out of the door every year.

I am half brained

In the spirit of Stumbling and Mumbling I have renamed my tagline to "Half-brained ramblings and insights...."

This is because I got called half brained by a guy doing a doctorate in Austrian economics when talking about sticky prices here.

Hmm I also got called Mr. Central Planner. Which is interesting. But I don't do intellectual name calling or baiting at all, and I won't even try and ridicule AJE's website at all because I quite like it actually, although admit to being a bit flaky about libertarian economics. I'm happy to learn under my own steam, but also happy to disagree or debate without the insults. Unless its insulting the royal family, politicians or Guardian columnists. Which are all fair game.

I did have a temporary feeling of intellectual inadequacy. I get this a lot when it concerns economics. The subject is so vast, subject to much theoretical debate and many ideas etc. Its actually quite overwhelming sometimes. I am happier sticking to the numbers sometimes. A spreadsheet or SPSS analysis can sometimes help answer more questions than theories from dead economists sometimes. Sh*t I must stop drinking 10 cups of black coffee a day. Does this induce paranoia?

For me, economics is all about learning. I would never call myself an expert at anything. As Kurt Vonnegut said about his table tennis playing days, just when he thought he was unbeatable, some young guy in the army came along and tore him a new asshole. Life is like that. But it doesn't dim my enthusiasm for learning more about economics and putting it into practice.

In terms of different models, theories, schools, I definitely do not stick to one philosophy. Maybe this is because I am also an athiest. I really view market models as imperfect tools for understanding how the economy works or doesn't. No different to most sensible economists I'll bet. I don't put so much stock in them as to remove common sense from operating from my full or half brain.

I am also a good analyst of data. I put a lot of stock in evidence too. This leads me to view theories as nice and vaguely useful, but not always relating to the numbers swimming around the screen in my spreadsheets. etc.

Does the US have STATE AID rules?

In my recent time in the US, it was notable that a lot of their economic development professionals work hard, and spend a lot of time and $$$ trying to lure mobile investment, and property developments into their patch. This is done with incentives, tax breaks, land deals, cofinancing, or tons of cash. Now - this led me to think - so they don't have State Aid rules then similar to the EU?

I have been advised that US guidelines are fairly weak. In all federal economic development programs, US states and local government authorities are barred from providing funds that lead directly to job displacement (i.e. moving jobs from one region to another). The purpose is to ensure that federal funds only support new job creation or job retention. Some (but not all) states have similar rules for intra-state job displacement. At the federal level, these rules are enforced in fairly strict manner. However, direct federal investments in economic development do not account for a very large part of activities in this field – it is mostly states and local governments who are active. Most of the money for economic development comes from states and localities and even then, the money comes from off-budget items like tax abatements as opposed to direct cash outlays. Like the EU, funds for worker training are exempt, and thus in the US it is possible to do the same things as were done in London with Ford. For recent new auto plants in Alabama and Mississippi, the major public investments were in the areas of training and local infrastructure (highway off-ramps, nearby roads, etc.). This money was primarily from the state, but the federal government did provide some workforce related funds.

Now this is interesting. But I believe that there is a good case for State Aid rules. Such incentives and deal brokering between localities can be a poor use of public cash. And its a pyhrric victory or zero sum game in the end in Europe.

However, anyone whose ever had to deal with State Aid issues knows its a f*cking nightmare (so I am told). First you ask the lawyers who tell you that it might be ok if an economist says there's market failure, and if you explain that you are an economist, they look puzzled... and so the dance begins.