From our Swindon Correspondent:
From the Guardian:-
PHE was responsible for collating the test results from public and private labs, and publishing the daily updates on case count and tests performed.
But the rapid development of the testing programme has meant that much of the work is still done manually, with individual labs sending PHE spreadsheets containing their results. Although the system has improved from the early days of the pandemic, when some of the work was performed with phone calls, pens and paper, it is still far from automated.
In this case, the Guardian understands, one lab had sent its daily test report to PHE in the form of a CSV file – the simplest possible database format, just a list of values separated by commas. That report was then loaded into Microsoft Excel, and the new tests at the bottom were added to the main database.
- How many cases were in the database before the update
- How many cases came in from each lab
- How many failures there were (e.g. validation problems)
- How many cases are now in the database.
I’m going to presume there are no controls in this process. Lashing Excel together is what some Johnny in a user department with no software development experience does, not experienced software designers. I’m guessing there’s no controls for missing files from a lab, duplicate data from a lab, malformed data from a lab, no audit trails, no testing process.
And there’s simply no excuse. The people with skills to do this aren’t cheap, but they also aren’t that expensive. There’s probably people in some parts of government on furlough who could do this properly. This is what you get from government.
The problem is that everyone has Excel on their work machines so they think is the easiest way to share data because then everyone has access to the data.
Interrogating data in a database requires some basic database knowledge so managers don’t like it as they aren’t in control. The daft thing is its fairly quick to learn and and there’s some good tools to make it easy and once a query has been created it can be run any time and the result imported in to Excel if they must have it that way.
Yeah. Even if you don’t want to learn SQL, there’s tools that allow you to build queries (like Power BI).
There are even database front ends that will present the data in a spreadysheety format so it looks familiar to the user.
Generally finance does a better job of this than everyone else, since that’s where the fines happen when things go wrong. HMRC employs and contracts work out to a lot of developers, good people. Maybe every other government department could learn.
Government used to have its own specialist internal computer consultancy – CCTA. This body had computer experts available for short-term loan to departments, and developed many of the standards used in the world today.
It was closed down in the 1990s on the grounds that outside industry could do the same job. Since she government computing has been a byword for expensively failed projects….
How (and why) are they clicking through the “WARNING: imported file has been truncated” dialog?
I work with electoral databases. There’s typically 70,000+ entries per constituency. After seeing the truncation warning once you think “oh yes, of course, doh!” and never do it again. Like trying to read a PDF with MSWord.
I think what you’re missing is that these people are not data experts, or even data novices. They will know nothing about it entirely. There’s a good chance the people running these offices will have been seconded from some other government or council department at a moments notice to get the illusion that ‘stuff is happening’ for politicians to crow about at press conferences. I have over the period of the lockdown been involved (as a spectator more than active participant) in a large planning permission application, one that itself involves over 1000 houses, that is part of an even… Read more »