2 February 2021

IT Solutions Architecture

Over the years I was in the role of IT Solutions Architect, (SA) but I don’t think I understood what was going on at a deeper level. I was an SA, for sure, and I did those things well.

I would be asked how to do SA. To me it was quite natural, so I didn’t ever find a satisfactory explanation to what I did know was in part an art form combined with technical skill.

I could describe in a way how I looked at things, how I would provide solutions that took a number of factors into account, express and document that, and help facilitate people to do that well.

I could use corporate terminology and describe how SA work reduces risks ensuring successful outcomes, therefore saving companies money. I could talk about providing a balance between what the company wanted, and how people work for their interests.

When people got stuck on what to do, I could outline the way forward to what would become a reality. I had some complex problems people could not solve.

But what was an SA? That’s where I felt some deeper level of dissatisfaction in trying to describe that. Of course I could give some stock answers to people who asked.

Now I understand more about what was going on.

There are at least two types of people in the corporate environment. There are those who just do things and do it with precision and outstanding ability – the technical people, or perhaps those of a conservative psychology. They can take what the creative people do and make it into something economically big. They think they do the work, so, to them, creative people are not working, even though our creative people are in fact working, just not the same way. There is no less energy involved. Those who were less creative, not the creative psychology, would try to put a box around those who were creative.

For instance, it really bugged one manager who wanted to make me do some formal course that would structure what I did. He put my work down. He was not intending to be bad, he just didn’t understand what was going on and the differences between two distinct classes of people.

We all know it takes a certain type of person to drive a train all day. If you put them into some other “box” you think is right, it will drive them crazy.

While an accountant may be great at what they do, they are happy to do what they do, but a creative person forced into the same style of work would go nuts.

It is not that one type of person is better than the other, but we lack a fundamental awareness of what is happening here.

Some would point out, this is very much tied up with biology, so that being the case, good luck with trying to make a creative person an accountant, or an accountant an artist. It will never work.

I think it comes to this simpler definition of creativity that supports the way the SA works. That is what I was not understanding, because I was just being me.

What has happened though, as in any area of human endeavour and formalisation, is that people move into roles of SA thinking they are the masters when they do not have that creativity. Then, the managers who are “conservative” do not understand creativity so they damage what the SA does. One example is trying to redefine what the SA will do in their organisation, or even removing their qualified job title to something else. This is almost without words to describe, but it is possibly ignorance, unethical and for personal gain.

People like the sound of things, so a number get into technical positions that present opportunities to be classified as an SA. However, the true definition of SA involves peers, even Internationally who recognise this role developing in someone’s work. It is a role that does not have classes and exams. It is entirely dependent on previous work history and a different level or way of working, which is why it is successful. This of course bugs those who have specific examinations and complex processes for their projects.

Part of the SA environment is highly technical, but it is also a role of much more freedom that regular employees experience. This is adverse to some, who may judge the SA and think they are trying to big note themselves, when they are not. The SA has a really valuable role. To dismiss it from some personal position of one’s own problems is nasty to say the least. It may also reflect strong immaturity of another person – which does happen. But yes, some people are jealous. The SA has a lot of freedom. I can talk to the shop floor operator with as much genuine concern and interaction as with a CEO, and am fully able to talk with both. There may be different dynamics at play too. Shop floor banter is different to the tricks technical mangers get up to, and they do try to trick. I found that out very quickly in Taiwan.

In the Taipei project, people did not leave their office to have lunch with each other, or have morning tea outside. I stopped a meeting and said I refused to continue. I said I was going outside to get morning tea and that someone would have to go with me or I would not be able to get back into the building. That is what I did. We also had a team lunch, which was something no one had experienced before, and would not again. People’s lives were not accomplishing anything more by having most of their lives living at their desks and sleeping under them. I was so desirous and glad to get out of Taiwan despite it having the best food in the world I ever had. Back to the point…

The project manager tested me without telling me he was doing that. He withheld information. At some point I said things did not make sense and why. I told him what should be happening for their processing to work. One could not know this without detailed training or information on both the software, and understanding processes and solutions. The manager and his offsider were very expressive with joy and excitement when I said this. They said they knew, that they were testing me, not telling me to see if I knew. This backhanded approach is not right, but as an SA one will have to deal with many types of situations which by the way only proves you are qualified, that you did not get into that position by some flaky way that people may want to think or redefine according to their own views. They cannot stand in front of a CEO and provide a solution either taking into account culture, position, or body language from those who make the power decisions. The SA role is not simplistic or purely gradable in some inappropriate test.

On that same project I attended a meeting with lots of diagrams on a whiteboard concerning the servers they had throughout Taiwan. This was not what we had come overseas for. It was out of scope. The manager asked me for a solution. I gave some conceptual ideas, and said I would be better having time to think it through and get back on it. He kind of laughed at me and said if I was a solutions architect I had to give a solution now! I stood up, went to the whiteboard and discussed a solution with some caveats. They were satisfied as it was a solution that could work. Whether they wanted to do it or could afford to given the complexity is another question.

I was always encouraged by two of my managers in the back of my mind and experience. Once, my manager said how I have to be more confident and not move into that mode of worrying and communicating hesitancy. After all, I was working with one of the largest companies in the world. That helped a lot. Another piece of advise from another manager during a project in Perth, was to just be me. Do what I like to do, who I am, and from that point things just fitted and went well. By the time I was on the Taiwan project I had confidence to meet those unexpected situations that came up. I look back on the timing of things in amazement. But the point here is that creative people only work well when the are in the flow. Flow is everything. If we stop and look at the cavern below us, or if we become self conscious, it stops the flow. It is a healthy experience to move in this flow and find solutions, mapping them out for others to move with us. People love it. This co-operative, creative process is incredibly strong for bringing people together under common goals and outcomes.

Awareness is part of the creative person. We think about things. Why are we doing what we do? How do I feel about that? What does it do to you? During the Taiwan project, I noticed the incumbent contractor was doing wrong things. During my summary in the CEO’s meeting room, I thanked them for having me, that I could not possibly understand all their business was doing, so I asked for their patience with me on that front. Then I said I would like to be fairly frank about a few things. Thus said, I moved on to detail. I showed how their contractor was telling them what to do rather than them telling the contractor what to do, and where they could change some fundamentals. Awareness lets us try to identify what is happening, and how to construct solutions.

Because a larger section of people are in a structured mode of working, they may not see solutions outside of the box or boxes they work in. Creative people move outside the box and validate what they do. They see that way. They can therefore see what you are doing, and gaps in the floor. Some highly intelligent and wonderful people may have our entire respect, but we may be asked some hard questions about why we approach some aspect of a solution in the way we do. This is a real challenge.

Our work is not pie in the sky – some conceptualisation. Our hands may knit together a solution just as much as any detailed programmer would. It is surprising how many people in IT do not want to be called a programmer. I have no idea why, as I would be proud to hold that title. It reflects a real and valuable capability. They deflect this terminology even though that is what they do. One problem with it is that programmers may build a “solution” that really annoys the end user, or has severe faults. You can say what you like, but no one will listen to you. These issues can’t be solved by intelligence or reason. It goes back to how someone is experiencing life, their emotions, their opinions, or other factors. You can participate in these projects, but not move past these sticking points. I once got so frustrated that a project was making clients go through hurdles in order to use the software. My manager asked me to pull back, to let them do what they were doing. He knew what was happening. I learnt a lot from that.

But my own technical skills, as part of the roadmap towards being an SA were developed, a lot actually. And those skills were under the banner of exams, testing, proven outcomes. On one project, the requirement was first a concept. We got into some details with the client. As we developed the design we could request the client modify parts of the design so that it may become programmable. Our solution also recognised the newness of the work, so we split the project into manageable components that were safe, that could be developed further as or if needed. A Business Analyst, or a Project Manager will not add this value in this way to a project. The final process was outlined enough for us all to work with, but no one knew how to do the algorithms to make it work. It was too complex for all those in the project and the company’s “programmers” basically refused to do the work. It took me months to work out the logic. My creative approach tells the story, which I understand better now. I would use pieces of blank paper, no lines on it, drawing many pages of diagrams and “equations”. This would take place at any hour of day or night. I had a notebook beside my bed. However, the final algorithms fell into place on some smaller sheets of paper when I was at the airport waiting for my flight. This is creativity under a longer period of time. The American Express project was a success.

I think it is a bit like playing music. If we get into some details, we know when a person is playing F# instead of F. This is how we understand things. I was able to participate with lovely people who were different to me with levels of skills and abilities to deliver on projects that put me in both awe and thanks. But what do you do if someone is partly autistic, or has some other fundamental experience of life that is not your own? How do we get tasks done with people who are not the same as us? This is about humanity and is very upsetting if we fail to understand and do. My saddest moment during one period of time in the workforce was when someone kept saying they would provide the details which I outlined for a project that would cost us immense penalties if we did not deliver on time. As we develop a project, many people become dependent on others making progress on time. But this one fellow did not work this way. To him everything would be all right. He would say it is under control. As we approached D-Day it was not. I offered ideas about taking notes on a piece of paper like a check list to get through the tasks. I had no other way to manage this. He did not work this way, but the complexity needed it. I always felt ashamed and upset because it was as if I was negating who that person was, who was an adult, a wonderful person. I had no choice. Clearly this project did not fit in with how he worked, therefore who he was as a person. I don’t think I will ever sit well with this even though I tried to be careful about it. The reality was the project would have failed. So there we have another example of how a project may make demands on people that we as people cannot deliver. I have seen those projects and the tears people have trying to execute them. I feel there are other ways, but corporations do not provide those other ways. At least I too know what it is to suffer in the workforce.

I was once told by a senior engineer that my solution for a project with Qantas would not work. This was not a side comment. It was forceful, direct, and repeated. I have been known to be determined, like a dog not letting go of a bone. I re-validated even more carefully what I was doing, but the engineer still did not agree – by which stage he stuck emotionally to opinion and pride rather than evidence. Of course, the project worked. And incidentally, there was no other known technical solution available or known at that point in time.

There was another project involving a transition of technology from what we call legacy to current technology. I knew the direction people were taking was not achievable. Technology has its own definitions, and no matter what you do, you can’t ignore that to peril. The person who proposed a way of dealing with the project, did not present a solution. He relied on one piece of software, that I knew could not do the job. He was the kind of person who went to sleep during the day in front of everyone at his desk. He did have some past contacts who got along well with him, but he moved into an area of work that frankly I think became his demise. So there was this oddity of being really liked and respected by those people, and considered with no delight, to put it mildly, in his present work.

I proposed the way forward, which involved work with a team in USA. We showed parts of the work working well, and how we would work with the additional data in the same way to complete what we could define as a set of work. Because this needed interaction between USA and Australia, and required time, we knew the project would work. It did. However, the project team was told from people higher up the time had to be sooner, and there was no choice. It did not matter to those people that on the day we go live, nothing would work, and people would have no choice but to impose fines and sack people. The team manager tried very hard to move to that piece of software to complete the project, but I persisted.

So, we have a number of factors at play during project development and execution. If you want to ignore your architects, so be it, you will suffer for it.

But what I found hard to deal with was people asking me how to do SA work so they could do it. They wanted those techniques. I could only talk informally about some things that make it work, but they still wanted some clear-cut formalisation to put into practice. Obviously, this is the difference again between creative and conservative.

But what remained a mystery to me for several years, was why were the architects in some companies doing what I instinctively knew was the right thing, and other companies had SA’s who were in some other world of their own. Was I misguided?

I learnt that ANZ Bank, for instance, had great architects. These were good people. We knew how to work together and get excited, producing results.

But the architects in an outsourcing company were a horror story. When I worked with them, my doctor said I would get clinically depressed if it went on any longer. I resigned and told the managers in no uncertain terms it was the worst job I had ever been in! And then some!

Their work was clearly not the SA role that I knew, but who was I to know different. Looking back, I believe their work was in conflict with the industry definition and processes for recognition of the SA role and its proven qualifications. But a company can do what they want to do, so there you have it. And some companies used the SA out of an industry requirement perspective, even if the people doing those jobs were more in my view technical architects as opposed to solutions architects.

An outsourcing company may have incredibly intelligent people who work non-stop with great complexities. There is no other way at times to handle the hardest projects that exist in Australia. That was not my joy, even though I loved complexity.

Some companies use their staff, knowing they are abusive. Their architects or projects leaders truly abuse other people in the fullest sense and get away with it, but those higher up know this gets what they want. Many times people walk in fear and try to hide from these shocking people on the floor. Who will recompense those abusive impacts on people’s emotions and lives? No one. Don’t think for a moment that Human Resources stands up to a wide range of bad people in corporations. This is not some rant or opinion, but is based on observable incidents over years of work within the workforce.

At the end of the day, there are many personality types and biological backgrounds we all have. This is not being fostered and protected to greater benefit in the workforce. Some managers see the differences, even if they don’t understand what is going on – they just know what direction to go in and what to support.

Today, many young people want good people working around them, good mangers. The abrasive and harmful people of course do exist, even savages with spears in that corporate jungle.

The lesson from this is that for so many years, I moved in and out of projects using all the skills I had, doing successful work, using diplomacy, psychology as required, but I never really knew the things I am mentioning here. It would have helped me when I suffered greatly at the hands of some others. I feel it really does come down to distinctive types of people, and that a good manager should recognise this in order to foster it properly. Without this, and with no accountability to such awareness, we work by methodologies in the workforce that have been prescribed as templates, and in doing so we stifle better outcomes and hurt people all the time. This is not a sob story. It is not crying. It is simply saying, hey, this is what is actually going on and it helps to know what is going on if we want to be more sane or do better together.

Better means a better bottom line, but you can be assured there are well meaning, good people with qualification who simply cannot make decisions in times of need, even though you explain how something will give the results. This total inability is at first astounding to watch, but it too comes back to who people are, rather than their job roles.

The solutions architect, the “proper” ones, mix creativity with practicality in a way that uplifts us all. Our minds work in a different way to those who then leverage our work and focus in those aspects of conservatism that hold structures together and produce economic gains. Both ends of the scale are necessary or we would not have biological diversity, or interesting societies and environments.

What is an example of this outside the IT realm?

Do you recall how ragged and tattered the feeling once was in West End, in Brisbane? It was really energetic moving in this area, or even going to the markets. Developers took over. More conservative people moved in. The area tried to retain its reputation, but it changed. Those beautiful things that once defined it were reshaped or moved out. Now it is known as a highly noisy and dense area.

Do you recall how Eat Street Markets were once a collection of spaces that let you feel you could explore and at times get lost. The new markets are formally laid out. That sense is permanently lost. The new markets are nice. I really like the formality, but it is not the same.

I think there is a continual interaction between creative and pragmatic, the most conservative staying safely in the middle of these environments, and the artists further out to the fringes. Both are necessary to the other but it is always changing.

People see something is their way, not yours, and you get undervalued, but it can flip too. We all know too well that the highly accomplished technical musician is not imparting the inner joy of a creative musician. The flipper here is that if we support what develops creativity, the technical levels actually become better, not the other way around. Think of that. Creative people know this. Our society’s advancements never take place in isolation from an energetic, improvised, creative environment and a formalised environment. Advancement in music needed both to happen, the local musicians bustling away with each other, improvising, playing at get togethers and concerts, and the pieces of paper that published the Beethoven Symphonies. Without that publication, music stagnated for centuries. And where did the printing press come from? I bet you it was seeded as a creative act. We need both improvisation, creativity, and formalisation to move forward. I think there is something worth thinking about here, even though I have not furhter developed these realisations or thoughts.

28 January 2022

Disaster Recovery for WordPress, and specifically with an EC2 Linux Instance

A DR Backup is used in emergencies where a standard backup fails to restore a system.

A system could use RAID disk to increases reliability, but not necessarily guarantee a restoration. We do not use RAID due to cost.

On Amazon, one can create a “snapshot” as a very efficient and fast way to recover from a point in time. However, if the snapshot copies an already corrupted service, there is a problem. It is possible to restore an entire system from a snapshot within 10 minutes. That is fantastic.

Companies design their disaster recovery backups. This suggests there is no complete method, but an approach to reduce risk of loss.

As individuals, we may also design a good backup. This differs to a regular day-to-day or standard backup.

As far as I know, a reliable backup for WordPress itself is the plugin called All-in-One WP Migration. This is a superb tool. There are others as my colleagues also use. But let us suppose such backups also fail.

Before we create a backup, if using Linux, we go to the root directory via our Shell login as root user and verify a full recursive listing does not freeze:

cd /
ls -laR

If the listing freezes, we may not have a good backup. We could try an Amazon snapshot, then restore to a new volume and see if that fixes the issue. If the “freeze” is occurring on WordPress files, it will likely mean a permanent failure, but you would have to use your own skills to examine further to see if those files can be removed without impact. For instance, a plugin can be deleted and removed from the database. There are cases I think, where there is corruption when writing to the database, but the database itself would not know the data was corrupt in and of itself. This gives you an idea of why there can be recovery problems we had not expected.

If a website has become “messy” over time and there is a failure we cannot pinpoint after extensive work, there may be all sorts of “stuff” in the database hanging around or some hard disk corruption. After all, hardware has historically been a bad enemy. We could build a fresh site, right from scratch, and simply use WordPress and plugin exports and imports, then fix the menus and widgets by comparing to a development or localhost copy. This does work and is not too hard, but you need some skill and effort to do so. Obviously if you have a https://mydomain.com that is live, you need to have a development system to rebuild to a fresh copy of https://mydomain.com. You don’t have to delete the old system, only disable it so you can go back to it if you forgot something, then wait until the new build is good and tested before deleting the old. You cannot run both systems on the same https://domain.com at the same time.

We approach the backup in various areas and decide ourselves our extent of work around them:

(1) The hard disk operating system and files – a “snapshot” for everything under the Linux root directory

(2) A Unix tar file for the website (e.g. /var/www/html) – for instance:

cd /var/www/html
tar cvf backup.tar ./* ./??*

Check the tar file can list the contents:

tar tvf backup.tar

Then you would download the tar file to your PC or wherever you wish, such as into the cloud.

Note that the WordPress media files may have been altered if you turned on compression with a plugin such as eWWW Image Optimizer. Your original files should be in tact somewhere else as uncompressed images as a matter of protocol.

WordPress introduced a maximum of 2560 pixel width on images and will scale them down like it or not. In my view this is really bad, but there may be settings in your WP Theme to change that limit, or use a plugin. Again, this points to the need to keep clean copies of your original media files somewhere else.

(3) Any special files you have stored, for instance under /home/user (e.g. /home/ec2-user). As an example, you may have some copies of shell scripts you use with crontab. Other files could be copies of httpd.conf, php.ini, www.conf, my.cnf, ssl.conf and various others such as 00-proxy.conf or phpMyAdmin.conf, /usr/share/phpMyAdmin/config.inc.php and so on. You simply use these files as reference, and would not copy them to a new installation. Also use “crontab -l” to list any crontab work you did and put it into a text file on your PC.

(4) The WordPress database – an SQL database backup.

This has issues around mysql versions that potentially cause imports to fail. We can check the database is not corrupted using an mysqlcheck command, but we have no database rollbacks available. We are not a corporation with expensive software. The all-in-one WP Migration plugin should be fine, but we can go further.

If using the all-in-one plugin, I would suggest temporarily deactivating plugins related to caching and security so we only have the plugins activated that we really need for a restore.

We can also use phpMyAdmin (or Unix database backup commands) to download the database, however, when we restore a database it sometimes fails. In that scenario it can help to have all the fundamental WordPress tables backed up (you select only those tables for an export) and then make additional backups for the other groups of plugins. This is still no guarantee. What often occurs is that a restore fails to load the menus and widgets. This is where it helps to have a localhost or development copy of a website (such as with the all-in-one plugin) so you can compare a restoration with a copy of an original. You may notice that a localhost copy uses instead of “localhost” in the wp-config.php file.

I am still old school and I prefer to stick with my database being latin1_swedish_ci. It causes me no problems.

There are some WP plugins that for whatever reason will sometimes not restore, even if you hunt around the forums for a fix and edit the database file. There is no easy way to discuss this so we have different angles at which we approach the DR backup in case one thing fails and there is another to go to. Consider this – you have a live retail website that a business depends on. It fails, for whatever reason, and you cannot restore it. This is a dangerous cliff face.

(5) You can backup WordPress pages/posts, media, plugins, and our theme – these are sets of files we Export (or write down manually). We also look for a WP Theme Export file and Global CSS settings.

(6) Written notes about how we did a number of configurations related to the operating system and WordPress – sometimes things get complex so we can’t remember how we did something. Keeping specific notes lets us rework a configuration from scratch or compare a new configuration to what we have previously done. We may have another website that uses the same configurations we can look at.

(7) Admin and User login details. Sometimes we forget this and it is problematic.

(8) Any operating system setups you need to refer back to, such as your DNS configurations, CDN, and so on. Think this way, in terms of Disaster Recovery. Imagine your whole system is gone, period. You have a blank slate and nothing to help you except your own files.

What are some typical things that get lost during a restore or transfer? Global CSS, navigation menus and widgets, and surprisingly, Sliders. This is where a development site comes to the rescue.

One may even make screen dumps to show the details for menus and widgets. However, I always, always make a text file backup using cut and paste of any widgets that use blocks, custom css or text. You just do not want to lose that.

This may all seem like a lot of extra work, but it is not your day-to-day backup, rather a disaster recovery backup for the unforeseen. It is up to you how much time you have, your skills, your level of risk comfort and reasonable liability. There are some folks who do not care at all, so it does not matter to them when they lose client data.

I would note that the plugin called “WP-Optimize – Clean, Compress, Cache” will be able to remove old copies of your web pages and posts, which greatly reduces the size of the database backup and restore. I like this, but would advise only to use this when a website’s current content is fully stable and you do not need reference to old pages. You can always take an important page and make it private.

Backups may fail when moving to a different database version. I have seen severe problems when moving from one web hosting provider to another because you have no control over the level of quality given by a provider. This is where it is really important to have a localhost or development system that allows one to compare during a rebuild. This can get tricky if you have to start editing the database yourself, but the all-in-one plugin I mentioned should remove these complexities.

Is it worth recording global CSS and theme settings? Yes. My experience is that at some point one may accidentally delete settings or make changes without pressing the SAVE button that are still saved! You need a precious copy of your CSS and theme settings. Each web page can usually have its own CSS settings too. A WordPress page (or post) export will keep these settings for you and can be viewed in an editor.

If you have any peculiar configurations, do keep a record. For example, you may modify a language file, or the permalink names for portfolios. Keep details on such things.

For day-to-day backups, we usually make a standard backup prior to minor software upgrades. I like using the snapshot method before significant changes.

At some point in our website administration, we will have site failures. This is a given. Even if we never upgrade software the hard disk could still corrupt files or a hacker could damage the service or WordPress, despite the security plugins etc. Even the hosting provider could issue a notice that either you or they need to move the site to new hardware, so a security upgrade is necessary.

DR is an approach and method that shows we have considered what we feel is sufficient. If we run into severe trouble, we are not in the same liable situation that would occur if we had not attempted to address DR backup as part of our work. We can demonstrate our DR approach as opposed to a designer acting dumb and saying they have no idea, that client data is lost!

Over the years in my IT work, whether a business or ASX listed company, I have seen backups fail. This is not the same as outages. Even preparations for various kinds of outages in data centers and companies have had incidents. For us, we are not building explosive proof building columns, hiring companies to take data tapes off-site, re-enforcing our roof structures to withstand a crash from a helicopter, using mirrored cascading computers in different geographic locations to manage aircraft landings, or diesel engines ready to supply power.

We ensure that our backups are derived from well protected sites. What is the point of delivering a system with concerns around lack of best practice? For example, why place a Forum database onto the same hardware space as WordPress? The security risk is too high.

As a note, hard disk failures are not likely to recover critical files when using Linux repair utilities. Many years ago things were pretty good using the fsck command, but these days other commands, for me anyway, have not restored working systems.

I have never lost a computer IT system that was literally in peril. But I personally have lost data and know that feeling. I have seen students lose their thesis or other lesser work and cry in front of the computer screen. These are real problems for today. I understand that loss. But my compassion is not quite justified when companies think they have installed secure backups, who then go merrily on their way and find the backups do not work in a crisis. True DR takes effort, testing, and maintenance in some way within those corporations.

I recall a power outage in North Sydney in the late 1990’s. The company had backup batteries, but the computer room was pitch black. There was no power in the batteries. Today we have other types of outages. For example, if our servers are not located in Australia, will the other country be exposed to typhoons? If our Internet service goes down, such as email failing for seven days, what would we have as backup to keep running? Again, in the 1990’s a famous Australian company almost lost it computer and data, for the entire company. I restored that system but was at a point in the restoration where I knew that when I pressed the ENTER key it would either fail completely, forever, or recover. I was in discussion with other professionals during the process. It recovered. During the outage, the company was able to use written books and ledgers to continue their business.

Another business, a hotel group, almost lost its data too. These critical situations were largely about insufficient disaster recovery backups and old IT systems. I cannot underestimate the need to keep reasonable pace with technology. If you are still stuck on WordPress version 3 !!! you have an exposure. If you are on PHP version5 or even today continuing on PHP 7.2 you have exposure.

Even highly regarded WordPress plugins can fail when they are upgraded, so a recovery is in terms of alternative plugins until the authors fix their plugins.

Another problem today is more important and obvious. That is the variable loads placed on systems. Most websites do not have this problem, but there are public sites of significant importance that have not built into them an architecture to sustain variable peak loads. To me this is not acceptable. Even some “lesser” websites have not been architected to meet reasonable public peak demand, so the sites are slow or freeze. Some sites may look good, but are they using caching tools and CDN? These are real problems. If your car sputtered at each set of lights or cross roads and you found ti difficult to drive off the mark, you would not put up with that. This however is exactly what a growing number of websites in the public domain are doing. I have no answer to this other than to say there are system architects who can design responsive systems, and that certain categories or segments of industry and government should be held to account to meet specific standards –  as there are no such standard in place at present. Imagine a service could put a label on their site to day it meets industry standard ABC level 1, or ABC level 2 and so on. Then we know where we stand.

As an example of tomfoolery, data centers must record metrix (data) on their outages, show how they address an outage problem with staff training and system changes, and pay penalties. These are real drivers. But, we find some entities advertising to the public that they have near 100% uptime and yet have no such processes in place, so the experience is that the services fail and people jump onto the forums to give their stories about excessive and painful downtime.

We need standards – but they are only met by those corporations who have a good IT culture, finances, architects, and best practices.