You may have heard about a Data Lake, but you might not have heard about a Data Swamp. Yes, they are a thing.
We all know that more and more data is being collected by companies. The need to collect increasing amounts of data and of course store it, risks creating data swamps. The word swamp came about as a way of describing the out of control, somewhat messy version of what was intended to be a data lake. I generally use the term for any scenario where the overall data repositories are messy, somewhat disorganised.
A data swamp is a horrible place, unlike a data lake which is a beautiful place that allows companies to retrieve and use their data efficiently and effectively.
The uglier, smelly, dangerous even, data swamps can make both those goals very difficult and perhaps impossible. They are a sign of mishandling data for sometimes years. They are a sign of poor data management and strategies.
However, all is not lost.
There are a number of strategies you can employ to ensure you don’t sink into the murky depths of a data swamp, which let’s face it, a number of those reading this will have one foot in already.
Data management strategy and governance
Ask yourself this. Do we have an actual over-arching data strategy? In response to this we have heard the following from landlords. Surely, we don’t need one if our systems are working ok. Surely that is for really large companies? Surely all we need to do is follow GDPR. All wrong, and indeed dangerous.
Excellent data governance is what equips your organisation to maintain a high level of data quality throughout the entire data lifecycle from creation to destruction.
Data governance defines how to work with data, who should access it, handle it, how long you retain the information for, deciding where the data is stored and so much more.
It simply isn’t enough to assume that these clever systems you have spent a lot of money on will do it for you. Sure, setting up roles and permissions in a system, in your active directory, and perhaps even making these the same thing isn’t enough. You need a strategy and indeed associated policies. You also need a strong governance model. They all go hand in hand so don’t scrimp. Get started, or get improving now, before it is too late.
Set yourself some homework to avoid a data swamp
You don’t get results from sitting back and doing nothing. A good strategy and policies ensure you have regular homework to do. If homework is a term that fills you with fear from your school days, then it’s easy, think of it as admin or maintenance. Maintenance is very apt as we all know that working parts need maintenance for them to continue to work at their best. Data is no different. It can very much be seen as a moving part in the operations of your company.
Tasks you would expect to do need to be formalised, structured and set owners, and those responsible for carrying them out. This is absolutely critical to the ongoing success of data management and turning your data into gold. One specific task that I wanted to highlight is that of cleaning your data. It links to the later area of data quality. If you don’t ensure your data is a clean or indeed tidy (not messy), then a great deal else falls into place neatly. Analytics make more sense, report and dashboards can be trusted and staff and customers are happier.
Reach for quality: having irrelevant data, old data, nonsensical data is simply not acceptable.
This is the cardinal sin of a great deal of companies across the world, in many sectors. It’s not something specific to social housing. We talk about rubbish in-rubbish-out. Well, I certainly mention this regularly. Imagine you need to make a decision based on the life span of a boiler part, but this information is messy, with some wrong dates in there. It may be that this was human error, or perhaps even the way your new system managed the data. The point here is that if you are not on it, and the data is not accurate, relevant or indeed even spelled wrong then decisions based on this data can end in disaster, literally.
The main take away here is that there is homework to be done. There is maintenance required, and if you don’t do it, and don’t set it out in the strategies and policies then expect a data swamp, expect mistakes, expect costs to rise and expect unhappy customers.
The next steps towards preventing a data swamp
So, above outlines three straight forward yet challenging areas to get sorted. To be fair, you can apply these to not just avoiding your data lake becoming a swamp, but you can simply use them as guides to better manage your data, lake or not. Need some help with that? Get in touch!