Leverage Machine Learning to Detect Insider Threats

Whether we’re talking about business or government, modern work practices require growing numbers of individuals to have access to critical organizational information. It’s a byproduct of this reality that insider threats, whether malicious or unintentional, are more frequently leading to severe and costly damages at large organizations. It takes only a quick news search to see that recent insider cases have led to the loss of export-controlled warfighting capabilities, the exposure of sensitive information, and even to attacks on U.S. military personnel. 

Insider threat programs are a natural and necessary response to these circumstances. They enable faster detection of potential risks and the ability to proactively respond to threats. But the growing complexity of risk behaviors and the large volume of data mining now required to effectively monitor them demands new and innovative technical approaches to the job. 

By adding advanced analytics and analysis to your organization’s insider threat detection and response toolkit, you can empower your security team to substantially improve monitoring, better protect assets, and more rapidly respond to risks and incidents. Booz Allen has helped advance insider threat tools and programs for government and commercial clients based in Washington, DC; Huntsville, AL; Chicago, IL; New York, NY; and elsewhere. We accomplish this work using machine learning (ML) algorithms in conjunction with the most recent major innovation in big data management: data mesh architecture. 

Originally conceived in 2019, the term data mesh refers to the decentralization and distribution of digital data at both the technical and organizational level. As an approach to managing and reaping insights from data, it embraces the fact that useful data is distributed in different forms throughout a given enterprise. Rather than requiring all collected data to be sent through a centralized processing apparatus before it can be accessed and put to various uses by various teams (including insider threat teams), data mesh is architected to let data remain with the domain-specific teams that harvest it while still making it accessible to end users—both technical and non—across the enterprise. This paradigm shift is an effective response to sticky challenges associated with scaling data accessibility and extracting value. With data mesh, business teams treat their own data as a product, meaning they're responsible for its quality and security. Distributed data pipelines—as opposed to a monolithic central pipeline—can respond and adapt with greater speed and agility, helping to avoid processing bottlenecks and technology antiquation. 

U.S. government insider threat programs generally focus on building tools that support analysts with data collection, alerts and monitoring, reporting, and case management. Machine learning algorithms can play a transformative role in all of the above. For example, they can proactively scan data sets to identify linkages between varying data points and pre-flag any anomalies. This frees human analysts from having to spend time on these tasks, allowing them to focus more on reviewing and processing potential threats. 

Machine learning can also advance insider threat mitigation by: 

• Implementing rule-based tests using quantitative and qualitative rules to detect specific threats
• Employing deep-learning and anomaly detection that can establish normal behavior patterns and recognize deviations from the baseline 
• Applying prediction modeling using supervised or unsupervised algorithms to train on historical data learning from past incidents
• Streamlining the data collection model to aggregate data and apply validated detection methods to measure risk across all domains

For an insider threat program to benefit from ML algorithms, first it must train and implement them. That often proves challenging. 

To succeed, machine learning algorithms must be trained against pre-collected, validated data sets. Collection, validation, and training all tend to be difficult and time-consuming. 

This is one of many areas where the benefits of data mesh come into play. Today, data collection happens continuously and at high volumes across a vast number of sources which must be governed and exposed to extract value. Data mesh architecture streamlines data collection and uses application programming interfaces (API) to expose data at the source, improving data accessibility, and supporting the reuse of data for a multitude of enterprise needs. While monitoring for insider threats, risk models managed as APIs expose data in real-time over known risk factors while data mesh architecture ensures data interoperability, standardization, and governance. On top of increasing data accessibility, data mesh improves analytical quality as well. It also has the ability to simplify and customize data pipelines, effectively decreasing analytical timelines.

Speeding data flow will remain key to advancing insider threat programs for the foreseeable future, and the transition to data mesh architecture is foundational to information acceleration. For insider threat monitoring software, machine learning integrated with data mesh architecture can help enhance end-user features by exposing trends, identifying emerging threats, and collecting metadata at an accelerated rate, building toward a repository for a single source of truth. Using original access controls, data mesh makes data sets available to any authorized application or user, thus fulfilling broader enterprise data sharing within and between organizations.  

Booz Allen has used API tooling and a “privacy by design” approach to cloud infrastructure to help a number of federal clients safely transition to data mesh architecture and improve scalability, security, and maintainability. Given that federal insider threat programs often act in coordination with other government offices, we are also helping clients find ways to securely and effectively distribute critical data assets across and between federal organizations. 

With the right strategy, technology, and implementation, combining machine intelligence algorithms with APIs and data mesh architecture helps detect potential risks faster. This gives organizations the ability to proactively respond to threats and empowers analysts to spend less time data sourcing and more time focused on preventing the harms that insider threats can cause.