Can data overload in the IC be tamed?

By Mark Wisinger, senior analyst

In an era with more sensors and data than ever before, how can intelligence agencies separate the significant data from the background noise?

Data management is an enormous challenge right now in the Department of Defense and Intelligence Community, especially as the internet of things emerges onto the scene. The big data challenges are linked hand-in-hand with cloud implementation, insider threats and cybersecurity, business intelligence and traditional IT infrastructure.

This data management challenge is an opportunity for big data vendors, with defense and intelligence agencies looking to the private sector for solutions in data access and search, data labeling and filtering and other management needs.

Here are some of the challenges and trends to watch for:

–Data hygiene and governance: As cloud has been encouraging the integration of information for analysis among the IC, data hygiene and governance has been a particularly onerous challenge. One of the big five has recently stood up a full-time data governance unit. When an agency devotes precious human capital to a problem, you know it’s significant.

–Multi-fabric, multi-domain environments: While the previous focus was on the Top Secret/Sensitive Compartmented Information level for shared services and cloud, the Office of the Director of National Intelligence is starting to take a look at the SECRET and unclassified spaces, particularly the possibility of a SECRET cloud. The Intel space is rapidly shifting to a multi-fabric, multi-domain environment. The data problem is only going to become increasingly compounded.

–Outdated architecture: As great as cloud is, there’s still plenty of “1990s” architecture (and older) in place. I’ve heard solutions providers talk about needing very specific environments for their solutions – that’s just not going to work in this space, especially when you go beyond just the DC-area locations. The field needs the same experience as the HQ.

–Defense data needs: DOD at large has been vocal as well about data management issues. STRATCOM has specifically asked for data access and search capabilities, for example. NORTHCOM needs help with data labeling and filtering. The Naval Criminal Investigative Service and the Air Force Office of Special Investigations have a significant need for data tagging and filtering tools.

These data hygiene requirements cut across the DOD and Intel sectors in pretty much every single branch, agency and component. And we fully expect big data will continue to be a technology focus in FY18 and beyond.

For more guidance on selling IT to government, reach out to immixGroup’s Market Intelligence team.