Using AI to classify ‘dark’ data
Written by Richard Hilton at Claritas Solutions
As businesses, we keep hold of vast quantities of data even when we believe they are no longer relevant. The information assets organisations create, collect, process and store during regular business activities only to archive offline in cold storage after a period of time. We fail to use this data again for new purposes and where it usually becomes difficult and costly to regain access to it. This is ‘dark data’.
The Jury is out on the real business value of the current version of generative AI, but one of the benefits it does have is to help your business classify old data. Enabling you to extract current relevant value to enhance your business information view in real-time and historically, as well as giving you the confidence to delete what data is not needed. A recent shocking statistic is only 15% of data is deemed business critical with 90% of data being deemed irrelevant 3 months post-creation.
Data that once may have been critical for a project or process, is archived or made ‘dark’ once it’s no longer valued in a business. Emails, PDFs, text documents, social media posts, call recordings and chat logs are put on to archive media, but more and more into public cloud cold storage. So does the cloud rely on us keeping poor data hygiene practices? Because the public cloud is built on the foundation of scalability – it accommodates it, but what cost is this having on organisations’ revenue and ultimately the planet?
Another point to consider is, that data that’s been archived may contain relevant information that could be impactful to your business today. However, because it is in an offline state you lose the insight from this data.
We often reinvent the wheel in business, redoing work anew for every project, which elongates a business’s reaction time and ultimately reduces productivity, increasing costs. There is also the problem that once data is offlined in an archive, you can guarantee that you will need it, skyrocketing your public cloud cold data costs due to the retrieval penalties.
This is where AI comes in. It can harness the capabilities of generative AI to capture and use metadata to classify your data giving you insight that is useful for you in your business today.
I work for Claritas Solutions, a Wetherby-based IT company and to me, it’s like rubbish dumping. We’ve been ‘dumping’ our data for years. On average, 50% of data stored is ROT (redundant, obsolete or trivial). Now we have a way to recycle it and add business value.
There are generally three data types – structured, unstructured and semi-structured. The way to use AI with dark data is to use and customise the metadata of files i.e. add a relevant reference and then save the file as an object. AI can then ‘read’ this metadata and organise it accordingly.
So instead of having shelves and shelves of non-organised boxes, each box has a label which can be read by an intelligent system that organises them in a meaningful way. It’s cleverly manipulating the data we already have.
AI can be used to search your whole business for useful information – not just data that’s in current use. Imagine a world where you could access all the information of your business, not just today but from more than 12 months ago and even across its lifespan. How would that change how you make decisions? Would it enlighten projects? Would it change your view?
How we use long-term data in a digital format can be revolutionised. Legal firms could use this for historical cases, the media could use it for articles, and historians could use it for archives. For other businesses, the data could help identify potential compliance issues or assess risks associated with certain business practices.
Amazon don’t get credit for many things, but one of the things they do incredibly well is use AI for their data. It means they can quickly adapt logistics to provide products that see increased demand and ultimately increase profit. They’ve done this more effectively than any company on the planet.
There are many ways this dark data can be utilised for valuable outcomes. Look at cancer data. We have already started to use AI to assess X-rays, but imagine this capability on a global scale, to track patterns and find new diagnoses for example? The possibilities for dark data are endless.
How many repeats are we doing? Look at what’s been done before and what’s been learned previously and quite often we don’t have to redo it. So let’s stop throwing away our data. Let’s be more considered. We can still use it for good and this is the possibility that AI brings.