technology

Excel and Track & Trace

The UK’s world class “Track & Trace” application “lost” 16,000 cases for over a week, as reported in the Register. Plenty of people have decided to comment and so I thought I’d join in and posted my thoughts in a linkedin blog, although I start this post with a quote from the Register.

The howls of disbelieving, horrified laughter caused by the news of the latest pandemic data cock-up yesterday were well deserved.
16,000 cases lost – purportedly in a blunder involving CSV data, row limits, and an out-of-date Excel file format? In a multibillion-pound, “world-beating” contact-tracing system? Unnoticed for a week of rising infection? In a system known to be broken for months but still not fixed?
Ridicule and despair, those shagged-out nags of our Johnsonian apocalypse, once again trudged exhaustedly across the plaguelands of England.
Rupeet Goodwins, The Register – 6th Oct 2020

Much has been made of the fact that the “dashboard” seems to have been implemented in an old version of Excel which has significant element array limitations and as pointed out to me by the Register, significant calculation errors which may lead to error program logic processing. This article talks a bit about why such decisions might be made and also how to perform good architectural practice and good program deployment and thus what might have been missing. It’s unlikely that such a mistake won’t be repeated, the people at the top, have not been through the painful process of failing in this way and paid a price, unlike many businesses. Once again, we know how to do this properly, not doing so is a choice based on ignorance or greed.

Any organisation of size needs an IT architecture plan. This maximises the opportunity for systems interoperability, compliance measurement within the supply chain, future scalability and cost control of both acquisition and support. Most architectural plans will include a spreadsheet as a desktop/user tool. Excel is the obvious and most popular tool, but reasons for worry, are expressed in their usual robust style, in this article, “Excel Hell: It’s not just blame for pandemic pandemonium being spread between the sheets”, from the Register, who also argue that Excel has no role in regulatory compliance software. However, at times Excel Basic has been the most popular development language in the world, and today, it still encourages a [structured] data driven analysis but it is poor for many to many relationships, graph semantics and list handling and MS are rebundling the tools, to charge more, for instance by issuing advanced functionality and new languages in Power BI. Using Excel is ubiquitous, and it fascinated me that this week, an article in efinacialcareers, bemoaned the lack of KDB & Q skills, identifying a lack of industry training as the cause because Universities won’t touch it because of the licencing costs, to be followed by a TES scoop about the end of the Union Learning Fund as the Govt. withdraws support, but just because it’s there, doesn’t mean you should use it.

Apart from architectural governance, the final piece of the jigsaw is the software development life cycle. A lot of effort has gone into understanding how to do this well with much written about requirements management of functional and non-functional qualities, testing of functional and non-functional qualities and release management. The processes of vendor and vulnerability management are also important parts of ensuring software does what’s needed safely.

Again, we, i.e. the industry know how to do this well, and even then there will be bugs. …

ARM in play again

I was interested to learn that ARM is in play again, although curious to learn that Nvidia might be its suitor, and even more interested to learn that Nvidia has overtaken Intel as the world’s largest chip fab. How did that happen? Nvidia sell on consoles as well as PC/laptops and games platforms are it seems another good whose demand has been boosted by CV19 and that the global demand for cycles has been driven by HPC and AI recently where the Nvidia are competing architecturally with Intel, although they need a CPU to complete their portfolio. It may be a better fit than I’d thought.

I have to laugh a bit, as Intel drove the final RISC players out of the market by leveraging the volume of the consumer product design, and it would seem, have been bitten in the arse by the same thing. These products require volume, and production will coalesce towards the low price duopoly.

ARM was bought by Softbank, for £24bn cash, just under 4 years ago; they are a Japanese venture/hedge fund which has famously had it its own problems. I wonder what they did with the money as some of their principals are now bleating for state protection as Nvidia is a allegedly an inappropriate owner of the chip designers. The Verge heralded it as another proof that intellectual property has value. The Register reports that the big stake holders have been insuring themselves against losing access to the intellectual property.

In this article on the BBC, they returned to Herman Hauser, one of ARM’s founders, who voted against the deal in 2016 who shares his fears for access to the technology of bought by another market participant, and possibly the decommitment to the Cambridge campus, which is a security of supply issue, but this Govt. is unlikely to do much and it should be safer owned by someone who wants the ideas rather than an organisation which just considers it a red-ink line in the P&L. …

There’s no divorce in Bitcoin

I attended a presentation hosted by the BCS, and given by Ron Ballard, based on his article in IT Now, “Blockchain: the facts and the fiction”. What he said inspired some thoughts and reminded me of others, some of which I have previously published on my blog. I wrote an article, called Learnings of Bitcoin, which was meant to be a spoof on the Borat film title and posted it on my linkedin blog, The article looks at the tight coupling of Bitcoin, and its consensus mechanism, the proof of work, together with its costs and vulnerabilities. It examines the goal of eliminating trust authorities and its questionable ability to meet the necessary roles of money as a means of exchange and a store of wealth. In the comment pushing it, I say, “This might be a bit basic for some, but you can’t have a coinless immutable blockchain, at least not one based on ‘proof of work’.”, at which point you need to consider if there are better data storage platforms for your use case.

Ballard is an anti-bitcoin polemicist and his anti-slide is comprehensive, although it misses the asynchronicity; I am less clear about his assertion that relational databases are good enough. Stonebraker in his paper, “The End of an Architectural Era (It’s Time for a Complete Rewrite) § 6.[12]” identifies five use cases where relational databases are sub-optimal including stream/feed processing and also states that SQL is not suitable for all data retrieval problems. I would add that an SQL database is not immutable, and most implementations do not come with a “four-eyes” super-user implementation; much effort needs to be invested in controlling the activities of these super-users, especially to implement the “developers can’t run code, and operators can’t change code” rule.

If the only use of Bitcoin, is paying for the electricity to keep it secure, exchanges become necessary. There’s something on how is Bitcoin valued by the market, but I haven’t found it yet also marginal utility theory requires two commodities; I wonder what the alternative to the crypto-currencies is and how liquid the exchange mechanisms are? Also are we trusting the exchanges? If so, the solution is not trustless! (I mean the alternative is cash since it’s the only way we measure energy and like cycles, electricity can’t be stored.

Some have suggested that the language used such as mining, is designed to suggest that the Bitcoin is similar and as safe as Gold, which isn’t! The use of the word ‘nonce’ which in IT security field means ‘number used once’, and is part of the chain design to my mind disguises the role of this field. It is only used once, but it is the output of the proof of work.

Ballard challenged his audience to find a good use case, and to my mind the proof of work and its costs make it hard, but I wonder if either trade confirmations or P2P name resolution are potential use cases, certainly bitcoin have had to solve the later and has inspired the namecoin project. There may be better ways to do both, but doesn’t look like anyone else is working on it.

The presentation was published on youtube here … …

delicious bookmarks recovered

I found my delicious bookmarks, and decided to remove the easily identified as gone away and expose the tags, on the way, I removed those without descriptions. The links came from from ~3525 to ~1800. I was fascinated by del.icio.us when I first discovered it, and used it as a micro blog. This is how it now looks in wordpress, but I need to put the anchor text, descriptions & tags through an ascii to html converter. The code is on github, in repo called delicious tools. The next stage is to allow them to be queried using the tags (or not, if I think it's worth it). The wordpress plugin on broken links continues to identify those that are broken, and I usually unlink them. If interested, 'read more' …

On Record Management

As part of my series on devising systems to create logs to protect an organisation and its staff against charges of criminality, I posted an article on my linkedin blog called “Doing Record Management well”. It doesn’t surprise me that there is an ISO Standard (ISO 15489) on the subject, but it does surprise me that I hadn’t heard of it until I started to research some of the articles in this series.

I have a research note on my wiki, which links to the Bank of England policy and also quotes Deutsche Bank’s policy, which is available because they post it on internet. I quote it here,

Deutsche Bank’s code of conduct, see page 25, says, among other things,

“Maintaining accurate books and records is fundamental to meeting our legal, regulatory and business requirements. You are responsible for maintaining accurate and complete records and for complying with all the controls and policies our bank has in place. You should never falsify any book, record or account that relates to the business of our bank, its customers, employees (including your own activities within our bank) or suppliers. You must never dispose of records or information that may be relevant to pending or threatened litigation or a regulatory proceeding unless you are authorised to do so by the Legal Department. You must also comply with applicable record retention policies.”
DB Code of Conduct

…

Knowledge Graphs

I attended a Capco/Semantic Web Company webinar, on Knowledge Graphs which provoked these thoughts, on how far we’ve come, new solutions to old problems and the social inhibitors to new technology adoption. The complexity of the data administration problem is why specialist tools have been developed and matured to the point that Gartner produce a Magic Quadrant on Meta Data Management tools, in which the Semantic Web company’s Pool Party appears as a visionary. The MQ report is currently being distributed, as is normal, by one of the “Leaders”, Informatica.

Andreas Blumaur, who was one of the speakers, repeated his suggestion, start small with committed users and that possibly the best 1st solution is a semantic search. (I thinl I’ll have another look at implementing something on my wiki.)

I have felt for a while that semantic web technology could be used to match work to resource in the cloud, with cloud entities advertising their capability using XML, it shouldnn’t be a stretch and with Azure, these systems are being defined in XML. The other application that interests me is if the XML/RDF models can be used to create a model of the person in the enterprise, maybe implemented in SQL; my current researches have not been fruitful. …

E2E & Zoom

The Zoom CEO stated at an Analysts Conference that they planned to introduce End to End Encryption (E2E) for their paying customers. At the moment, zoom does not do E2E encryption, they are encrypted between the user device and Zoom’s servers, but zoom’s servers can be tapped. This means that GCHQ can’t see what’s happening, but the NSA & FBI can. (This assumes that GCHQ can’t break properly configured TLS.) In the end, doing zoom rather than skype or google hangouts, if you believe them to be more secure, is like going to a meeting and trying to spot the special branch cop, preferably before you’ve fucked them. The rest of this blog discusses the issues of the device security, technical complexity, and the problem of user identity. See below/overleaf … …

Can’t make it up

A note on LinkedIn on why managements need IT usage policies to prove their compliance and to act legally and fairly towards their employees. I suggest that ISO27001 is useful as a technical standard and COBIT as an organisational one.

This was written in the light of a couple of cases I had to deal with as an accompanying rep. or as an advisor.

You can’t claim that users are not performing if you can’t prove the IT systems work as documented. You can’t pursue a conduct disciplinary against people operating a policy. You can’t fulfil FOI or SAR requests if the data retention policy is suspect. You can’t be sure that corruption has not occurred if there is inadequate segregation of duties.

Having policy will help the organisation answer the following questions. Is our software supported? Why and how was that data deleted? What should be logged? Who has permission to read, amend and run these programs and/or this data? Are our vendors signed up to our IT security goals? Why do you not know this?

This is all defined in these standards, and the GDPR makes certification to good practice evidence of good will. ISO27001 and COBIT are the big boys in town to prove technical and organisational protection.

You can’t make it up anymore. …

Theory matters!

I have just posted a blog on linkedin about business and IT strategy. I say a bit more here! This was provoked because I was doing some research for a job application which involves IT strategy. I was considering the alignment of business strategy with that of the IT department and what I might say. I outlined three models, although they were all developed a while ago, I think they all have relevance today. The three models address business strategy, software portfolio management and architectural pattern selection. Business strategy should drive portfolio and project management choices. While business strategy will outline how to do what must be done, it also defines what will not be done. Portfolio management determines the allocation of development funding, priority, maintenance funding, project risk appetite, people skills, project governance and software sourcing policy and as result of choices made, one can select the appropriate platform super architectures, of which you may need more than one. I conclude that theory matters. See more below/overleaf … …

More on Free broadband

One of the allegedly glaringly popular promises in the last Labour manifesto was the promise of free broadband. I wrote about it a couple of times and decided that though it was desirable, the failure to socialise it both within the movement and society as a whole allowed the promise to be undermined by questions of cost. I concluded the article above by asking “why not free water?”. One of the answers to that is that much government business, which it requires those most likely to not have the internet requires its use, on the one hand, recording business activity and on the other claiming benefits and maybe equally importantly writing to one’s MP or Council. The idea came from two sources, one of them the London CWU who issued a pamphlet, calling for broader public ownership of the telco and postal services on the grounds that private/market ownership was wasteful of wealth and resources, has delayed the adoption of superfast broadband, caused an investment famine and led to a pursuit of short term fashion & profit rather than investing in infrastructure for long term aggregate growth. The Executive Summary is reproduced below/overleaf. … …

« Previous