Building Pipelines with PassiveTotal

One of the most powerful features inside of PassiveTotal is the ability to monitor infrastructure and receive alerts when something changes. We’ve covered how to deploy monitors in previous postings and videos, but never showed how they could be used for follow-up actions. By combining the notifications and monitors API from the account endpoints, it’s easy to create an automated system to block or publish threat data.

Crimeware Focus

Over the past few weeks, we've taken the time to focus some more time on BePush analysis. If you aren’t familiar with the group, it’s a small group of users based in Turkey that abuse the Facebook platform through malicious creatives in order to generate ad-based clicks. What makes this group a good candidate for automation is the ratio of domains to IP addresses. BePush actors tend to invest in purchasing new domains, but seldom move their core infrastructure unless it’s really impacted. Our primary goal with this project was to automate the discovery and reporting cycle for known BePush infrastructure.

Automated Discovery

In order to generate alerts in PassiveTotal, we must first monitor some infrastructure. Through some public postings and online analysis, we were able to find several active IP addresses being used for command and control. If you’re interested in seeing how monitors work in action, consider monitoring one or all the following: 5.196.91.128, 139.129.118.232, 188.214.128.222, 188.166.237.28.

We’ve had monitors set on these IP addresses and have been getting alerts on a daily basis when new domains come online. Here’s an example of the most recent monitor alert for 5.196.91.128.

The output of this monitor is straight-forward and easy to process. Two new domains associated with this IP address (grildddwqdred.xyz and www.haberleringundemi.com) and while they are not guaranteed to be malicious, given the history of this IP address and active campaigns, we want to automatically process them. Now, using the web interface is one way to get this data, but for the sake of automating this fully, we are going to want to use the notifications API.

Gathering Alerts

In order to get the specific notifications related to BePush, we need to first ask PassiveTotal for a list of our monitored infrastructure. This can be done using the monitors API which returns not only a list of items, but also tags and their last alert time. Since all we care about is BePush, we can simply check the tags array for the “bepush” tag and move on if there isn’t a match.

The above screenshot shows a couple lines of code that uses the PassiveTotal python library in order to interact with the API. Each of the monitors is processed, checked for the proper tag and then all notifications are requested.

Processing the Results

With code in place to get all the BePush-related notifications, we can now move on to the processing stage. Keeping in mind our goal to automate the analysis cycle, we want to tag and classify any domains associating to active BePush infrastructure accordingly. Doing this means we continue to build a body of knowledge on BePush and can then use our other API endpoints in order to feed systems for blocking or defense.

Also, as Facebook ThreatExchange members/integrators, we are keen on sharing this information back out with a larger audience. Using the python bindings for ThreatExchange, we can create Threat Descriptors that classify and describe the infrastructure used by BePush while also preserving the context of the monitor hit.

Monitors for BePush tend to alert on a daily basis. In order to avoid re-processing all the notifications, we define a period of the past two days for processing. Additionally, we only really care about passive DNS hits, so if the alert is not part of that category then we skip the result. Assuming all conditions are met, we ship the notification to some code to process the Facebook ThreatExchange posting and then tag and classify the value using the PassiveTotal actions class. To see the completed script, check out the following Gist and be mindful of the variables up top for the configuration.

As a last and final step, we need to make sure this script runs every day without user interaction. Using something like cron or celery, it’s easy to schedule this script to run as often as you’d like.

Building your Own

Creating the pipeline for BePush infrastructure took less than 30 minutes using the PassiveTotal APIs and it will continue to save us time literally every day it runs and processes BePush hits. There’s a lot of room for improvement in our sample, but we wanted to show how easy it was to get started building your own pipeline. Additionally, we showed how easy it was to begin interacting with Facebook's ThreatExchange. All of the data PassiveTotal shares on BePush will be marked as TLP:WHITE, so the entire community can take advantage of blocking this threat.

Even if you are not a developer, we encourage you to try building out your own pipelines using PassiveTotal's API. Use our code samples as a reference and if you get stuck, have any questions or just want some help, shoot us a message at feedback@passivetotal.org.