Organizations seeking new ways to meet the challenge of cybersecurity, protecting vulnerable online assets from attack, are looking towards a new generation of solutions incorporating artificial intelligence and building on strategic open source software projects. These approaches are incorporating machine learning and open models to examine enormous volumes of log data in order to discern patterns and make predictions, in an effort to get ahead of attackers.
This week saw the strategic partnership of Versive, provider of AI-powered cybersecurity, with Cloudera, provider of a platform for machine learning and advanced analytics built on the latest open source technologies.
The joint solution enables organizations to fill the most urgent enterprise cybersecurity gap: the ability to automatically detect advanced persistent cyber threats across the network. The partnership combines Versive’s advanced AI technology with Cloudera’s advanced analytics and machine learning capabilities to create an enterprise-scale, automated, adversary detection solution.
“Cloudera has established itself as the premier provider of enterprise-grade, big-data deployments,” stated Versive CEO Joe Polverari in a press release. “On the foundation of Apache Hadoop, Spark, and Spot, Versive is revolutionizing how enterprises address the most dangerous, persistent cyber threats by automatically detecting previously invisible, high-risk threats that no other tool can see. Our vision is that every corporate security organization will be powered by advanced AI, and together Versive and Cloudera are the foundation.”
The Versive Security Engine (VSE), driven by Cloudera’s core infrastructure, is built on an extensible platform designed to systematically automate and enhance human expertise with advanced AI, in order to detect never-before-seen threats. Versive is a software-only solution built on open-source technologies.
The VSE can detect adversaries that have already slipped past perimeter defense; it connects threat behaviors to surface the adversary campaign, finding the thread through disparate alerts; and it can deliver actionable threat cases that warrant immediate investigation.
Cloudera Enterprise built on Apache Hadoop delivers a platform for data management and analytics. Cloudera stores, processes and analyzes any data volume or type, while enabling machine learning and ad hoc queries. Enterprise uses Open Data Models to separate data from applications, to provide maximum flexibility for analysis, and placing the customer in control of their information.
“The key to an effective modern security strategy is to achieve comprehensive enterprise visibility by centralizing your sensor, telemetry, and context data, and using sophisticated artificial intelligence to make sense of it all. Using the Cloudera platform, Apache Spot’s open-data models, and the Versive Security Engine, enterprises can detect attackers that would be unseen with other approaches,” stated Sam Heywood, Director of Cloudera Cybersecurity Strategy, in a press release.
This partnership marks the beginning of a transformational shift in how advanced AI is used for cybersecurity in the enterprise.
Asked in an interview to elaborate on the importance of AI to Versive’s offering, Versive’s VP of Engineering Dustin Hillard said, “Today’s dynamic cyber threat environment is overloaded with corporate network defenders.” According to Ponemon Institute, organizations can get 17,000 alerts a week, with only 19% of the alerts considered reliable and only 4% ultimately investigated. “The network defenders are overloaded with data. The use of AI allows them to prioritize true adversary campaigns and whittle through the noise of their traditional SOC environment. Versive’s AI platform is designed to map to adversary campaigns to give network defenders a fighting chance,” Hillard said.
When Versive starts an engagement, they peruse logs going back weeks or months to gather data. It helps if the organization is a user of the Cloudera platform, with its Open Data Model.
“Breaches from advanced adversaries average over 200 days. We scan weeks or months of data, and produce less than 10 cases per week on average,” for a client to investigate, Hillard said. “Establishing a long-term time frame is a key aspect to our approach.” With its software-only offering, Versive can also scale to any size data a client has; no need to add additional appliances for instance.
Versive is just coming to market with its offering. It has been deployed in two production environments so far, both in financial services, for a few weeks. Each environment has over 100,000 nodes under analysis.
The experience has been positive so far. “Initial customer deployments have successfully detected red team activity, potential insider activity, and unapproved data movement,” Hillard said. “Identified threats also help prioritize continued improvement of security instrumentation and tooling.” [Editor’s note:A red team or red force is an independent group that challenges an organization to improve its effectiveness.]
Cloudera’s open-source Apache Hadoop distribution, CDH (Cloudera Distribution Including Apache Hadoop), targets enterprise-class deployments of that technology. “Customers are trying to extend the amount of data they are analyzing,” said TJ Laher, in cybersecurity marketing for Cloudera, in an interview. “We are the foundation data management and advanced data analytics platform to support those efforts.”
“If the company’s log data has already been ingested into the Open Data Model, it accelerates the process,” Laher said. “So a key part of how we see this approach becoming easier and easier, is that as the basic schema is adopted, customers can go quickly to the analytics framework. This is as opposed to many organizations that experience big data applications that are difficult to get off the ground and keep running.”
Learn more about the combination of Versive and Cloudera.
Some History: From SOCs to Spot
The cybersecurity journey to the present has been an evolution. Security Operations Centers (SOCs), a centralized capability to handle security incidents across thousands of endpoints, emerged in the 2005-2008 time frame and have been adopted by many organizations. For example, the US Transportation Security Administration has implemented SOCs as a security communication hub for many airports.
SOCs provide tools for data collection, data aggregation, threat detection, advanced analytics and workflow capability from a single management area. The advent of big data in 2008 or so caused too much data being sent through the SOCs, overloading them. Cisco’s OpenSOC was an open source project started in 2013, with Apache Hadoop underlying it for data analysis, aimed at helping detect security threats and hack attempts more effectively. That project gave birth to multiple others include Apache Metron and Apache Spot.
Apache Spot, formerly Open Network Insight, was originally developed by Intel Corp. with a focus on building a big data analytics platform, with machine learning capabilities for cybersecurity use cases. In September 2016, Cloudera and Intel together donated this community-driven open data model to the Apache Software Foundation as an open source project. As an incubator project, it got new life and was renamed to Apache Spot. It is fundamentally based on the Cloudera platform.
Spot expedites threat detection, investigation, and remediation via machine learning and consolidates all enterprise security data into a comprehensive IT telemetry hub based on open data models. Spot’s scalability and machine learning capabilities support an ecosystem of ML-based applications that can run simultaneously on a single, shared, enriched data set to provide organizations with maximum analytic flexibility. Spot harnesses a diverse community of expertise from Centrify, Cloudera, Cybraics, Endgame, Intel, JASK, Streamsets, and Webroot.
Apache Spot uses machine learning as a filter for separating bad traffic from benign and to characterize the unique behavior of network traffic. A proven process, of context enrichment, noise filtering, whitelisting and heuristics, is also applied to network data to produce a shortlist of most likely security threats.
The community of cybersecurity approaches incorporating AI is expanding into an application ecosystem. Here we examine a selection of the players.
Cybraics uses AI to be Proactive in Cybersecurity
Cybraics is a security analytics and artificial intelligence company offering nLighten, delivered as software-as-a-service. NLighten combines multiple modes of machine learning with an advanced AI engine to find unknown, advanced and insider threats.
Cybraics recently added chief technology officer Alan Ross, who has been chief cloud security architect at Intel, and who led Intel’s IT security architecture and technology development. He is also a pioneer behind the Apache Spot program, aimed at providing tools and best practices for building out a big data security analytics platform.
The home page for Cybraics reads like a call to arms:
“At Cybraics, we believe that cybercrime is the greatest threat in modern history.
Every day hackers, terrorists, and nation states are looking for new ways to infiltrate and attack corporations around the world. And every day, they are finding ways around our defenses, costing corporations and consumers billions of dollars.
The problem is the current security paradigm is flawed; it is based on companies learning from events that have already happened. This leads to a rate of learning that is far less than that of our adversaries. It forces us into a reactive position, rather than a proactive posture, and that leads to the well-known adage that criminals are always “one step ahead.”
We believe that it is time to finally remove the “edge” our attackers have over us and to find a new approach to thwart their attacks. In order to fight our adversaries, we must increase our rate of learning. We must learn from them in a proactive way. We must learn from their actions, not their outcomes.”
Learn more about Cybraics.
Endgame Has Roots in Intelligence Agencies
Endgame offers the Hunt platform, aimed at blocking and removing threats at the earliest stages.
In a recent test from AV Comparatives, Endgame’s platform achieved a 99.5% protection rate against attacks mirroring those enterprises encounter daily. Andreas Clementi, CEO of AV-Comparatives, stated in a press release, “Endgame performed very creditably on their first anti-malware test, both in blocking malware and avoiding false positives. It’s more than a match for existing AV products.”
Endgame also recently joined the Anti-Malware Testing Standards Organization. And in a test by SE Labs, Endgame’s machine-learning malware prevention engine achieved a 100% anti-malware effectiveness rating.
Endgame was started in 2008 by Chris Rouland and other executives who previously worked with the CIA. Endgame’s current team includes several veterans of the National Security Agency (NSA) and Department of Defense. Before joining Endgame, Jamie Butler, CTO, was a computer scientist at the NSA and later chief researcher at Mandiant, and chief architect at FireEye. He concentrated on research advanced threats, vulnerabilities and attack patterns.
Tony Meehan, VP engineering at Endgame, leads development of the company’s cyber operations platform. He spent nine years at the NSA, where he served as a technical director and chairman of the Computer Network Exploitation Developer’s conference.
Amanda Rousseau, research engineer, concentrates on malware. She worked for two years at the Department of Defense Cyber Crime Center as a malware reverse engineering and computer forensic examiner.
Learn more about Endgame.
JASK Announces Trident at Recent Black Hat Conference
JASK Trident, an AI-powered security operations platform, was announced recently at Black Hat in Las Vegas. Trident aims to speed up the process of identifying attacks. It does this by ingesting a large volume of disparate data, comparing it to known attack indicators and patterns, and refining the data down to key alerts. The massively reduced set of alerts is then managed by security analysts. The time it takes is thus reduced from days to hours.
JASK described the functionality of Trident in a press release as:
- Monitors networks end to end, surfacing and triaging the most relevant attacks using advanced AI, while providing a clear picture of the attack surface.
- Applies machine learning-based analytics to detect potential malicious behaviors by assets and users across the network.
- Offers modern ad hoc data exploration and visualization capabilities through “notebooks.”
- Allows security analysts to configure any external and internal context enrichment that operationalizes data aggregation to dramatically reduce time to insight.
JASK is available as a cloud-based solution.
Learn more about JASK.