mirror of
https://github.com/viq/NewsBlur.git
synced 2025-04-13 09:38:09 +00:00
Blog post: How a Docker footgun led to a vandal deleting NewsBlur's MongoDB database
This commit is contained in:
Samuel Clay
parent
0e81d2f346
commit
4b51adb691
9 changed files with 312 additions and 153 deletions
|
|
@ -1,89 +1,129 @@
|
|||
---
|
||||
layout: post
|
||||
title: Story of a Hacking
|
||||
title: How a Docker footgun led to a vandal deleting NewsBlur's MongoDB database
|
||||
tags: ['backend']
|
||||
---
|
||||
|
||||
I'd like to answer a few questions about what happened here.
|
||||
I'm in the process of moving everything on NewsBlur over to Docker containers in prep for a [big redesign launching next week](https://beta.newsblur.com). It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into [a new book on how to tame the machines once they're smarter than us](https://en.wikipedia.org/wiki/Human_Compatible) when I received a strange NewsBlur error on my phone.
|
||||
|
||||
"query killed during yield: renamed collection 'newsblur.feed_icons' to 'newsblur.system.drop.1624498448i220t-1.feed_icons'"
|
||||
|
||||
There are honestly no sets of words in that error message that I ever want to see again. What is the word `drop` doing in that error message? Better go find out.
|
||||
|
||||
Logging into the MongoDB machine to check out what state the DB is in and I come across the following...
|
||||
|
||||
{% highlight javascript %}
|
||||
nbset:PRIMARY> show dbs
|
||||
READ__ME_TO_RECOVER_YOUR_DATA 0.000GB
|
||||
newsblur 0.718GB
|
||||
|
||||
nbset:PRIMARY> use READ__ME_TO_RECOVER_YOUR_DATA
|
||||
switched to db READ__ME_TO_RECOVER_YOUR_DATA
|
||||
|
||||
nbset:PRIMARY> db.README.find()
|
||||
{
|
||||
"_id" : ObjectId("60d3e112ac48d82047aab95d"),
|
||||
"content" : "All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump."
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
Two thoughts immediately occured:
|
||||
|
||||
1. Thank goodness I have some recently checked backups on hand
|
||||
2. No way they have that data without me noticing
|
||||
|
||||
Three and a half hours before this happened, I switched the MongoDB cluster over to the new servers. When I did that, I shut down the original primary in order to delete it in a few days when all was well. And thank goodness I did that as it came in handy a few hours later. Knowing this, I realized that the hacker could not have taken all that data in so little time.
|
||||
|
||||
With that in mind, I'd like to answer a few questions about what happened here.
|
||||
|
||||
1. Was any data leaked during the hack? How do you know?
|
||||
2. How did NewsBlur's MongoDB server get hacked?
|
||||
3. What will happen to ensure this doesn't happen again?
|
||||
|
||||
Let's start by talking about the importance of your data. As you may know, NewsBlur is open source and enjoys the added protection of having thousands of people looking at the codebase and dozens contributing back to it.
|
||||
Let's start by talking about the most important question of all which is what happened to your data.
|
||||
|
||||
### 1. Was any data leaked during the hack? How do you know?
|
||||
|
||||
I can definitively write that no data was leaked during the hack. I know this because of two different sets of logs showing that the automated attacker only issued deletion commands and did not transfer any data off of the MongoDB server.
|
||||
|
||||
This is what the day looks like. This 24 graph of bandwidth shows:
|
||||
Below is a snapshot of the bandwidth of the db-mongo1 machine over 24 hours:
|
||||
|
||||
<img src="/assets/hack-timeline.png" style="border: 1px solid rgba(0,0,0,0.1);">
|
||||
|
||||
You can imagine the stress I experienced in the forty minutes between 9:35p, when the hack began, and 10:15p, when the fresh backup snapshot was identified and put into gear.
|
||||
You can imagine the stress I experienced in the forty minutes between 9:35p, when the hack began, and 10:15p, when the fresh backup snapshot was identified and put into gear. Let's breakdown each moment:
|
||||
|
||||
1. **6:10p**:
|
||||
2. **9:35p**:
|
||||
3. **10:15p**:
|
||||
4. **3:00a**:
|
||||
5. **4:30a**:
|
||||
1. **6:10p**: The new db-mongo1 server was put into rotation as the new MongoDB primary server. This machine was the first of the new, soon-to-be private cloud.
|
||||
2. **9:35p**: Three hours later an automated hacking attempt opened a connection to the db-mongo1 server and immediately dropped the database. Downtime ensued.
|
||||
3. **10:15p**: Before the former primary server could be placed into rotation, a snapshot of the server was made to ensure the backup would not delete itself upon reconnection. This cost a few hours of downtime, but saved nearly 18 hours of a day's data by not forcing me to go into the daily backup archive.
|
||||
4. **3:00a**: Snapshot completes, replication from original primary server to new db-mongo1 begins. What you see in the next hour and a half is what the transfer of the DB looks like in terms of bandwidth.
|
||||
5. **4:30a**: Replication, which is inbound from the old primary server, completes, and now replication begins outbound on the new secondaries. NewsBlur is now back up.
|
||||
|
||||
The most important bit of information the above chart shows us is what a full database transfer looks like in terms of bandwidth. From 6p to 9:30p, the amount of data was the expected amount from a working primary server with multiple secondaries syncing to it. At 3a, you'll see an enormous amount of data transfered.
|
||||
|
||||
This tells us that the hacker was an automated digital vandal rather than a concerted hacking attempt. And if we were to pay the ransom, it wouldn't do anything because the vandals don't have the data and have nothing to release.
|
||||
|
||||
While the server was being snapshot, I used that time to figure out how the hacker got in.
|
||||
|
||||
### 2. How did NewsBlur's MongoDB server get hacked?
|
||||
|
||||
It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun.
|
||||
Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn't work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was "active", doing a `sudo iptables -L | grep 27017` showed that MongoDB was open the world.
|
||||
|
||||
nbset:PRIMARY> show dbs
|
||||
READ__ME_TO_RECOVER_YOUR_DATA 0.000GB
|
||||
admin 0.000GB
|
||||
local 16.471GB
|
||||
newsblur 0.718GB
|
||||
To be honest, I'm a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlur's MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored. And the Docker-made hole was immediately patched.
|
||||
|
||||
nbset:PRIMARY> use READ__ME_TO_RECOVER_YOUR_DATA
|
||||
switched to db READ__ME_TO_RECOVER_YOUR_DATA
|
||||
It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security. Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.
|
||||
|
||||
nbset:PRIMARY> show collections
|
||||
README
|
||||
system.profile
|
||||
<img src="/assets/ornament-pill.png" style="display: block; margin: 0 auto;width: 100px;">
|
||||
|
||||
nbset:PRIMARY> db.README.find()
|
||||
{ "_id" : ObjectId("60d3e112ac48d82047aab95d"), "content" : "All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump." }
|
||||
The second reason we know that no data was taken comes from looking through the MongoDB access logs. With these rich and verbose logging sources we can invoke a pretty neat command to find everybody who is not one of the 100 known NewsBlur machines that has accessed MongoDB.
|
||||
|
||||
Looking at the MongoDB access logs, we can invoke a pretty neat command to find everybody who is not one of the 100 known NewsBlur machines that has accessed MongoDB.
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight" style="max-height: 200px;"><code>
|
||||
$ cat /var/log/mongodb/mongod.log | egrep -v "159.65.XX.XX|161.89.XX.XX|<< SNIP: A hundred more servers >>"
|
||||
|
||||
$ cat /var/log/mongodb/mongod.log | egrep -v "159.65.XX.XX|161.89.XX.XX|<< SNIP: A hundred more servers >>"
|
||||
2021-06-24T01:33:45.531+0000 I NETWORK [listener] connection accepted from 171.25.193.78:26003 #63455699 (1189 connections now open)
|
||||
2021-06-24T01:33:45.635+0000 I NETWORK [conn63455699] received client metadata from 171.25.193.78:26003 conn63455699: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
2021-06-24T01:33:46.010+0000 I NETWORK [listener] connection accepted from 171.25.193.78:26557 #63455724 (1189 connections now open)
|
||||
2021-06-24T01:33:46.092+0000 I NETWORK [conn63455724] received client metadata from 171.25.193.78:26557 conn63455724: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
2021-06-24T01:33:46.500+0000 I NETWORK [conn63455724] end connection 171.25.193.78:26557 (1198 connections now open)
|
||||
2021-06-24T01:33:46.533+0000 I NETWORK [conn63455699] end connection 171.25.193.78:26003 (1200 connections now open)
|
||||
2021-06-24T01:34:06.533+0000 I NETWORK [listener] connection accepted from 185.220.101.6:10056 #63456621 (1266 connections now open)
|
||||
2021-06-24T01:34:06.627+0000 I NETWORK [conn63456621] received client metadata from 185.220.101.6:10056 conn63456621: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
2021-06-24T01:34:06.890+0000 I NETWORK [listener] connection accepted from 185.220.101.6:21642 #63456637 (1264 connections now open)
|
||||
2021-06-24T01:34:06.962+0000 I NETWORK [conn63456637] received client metadata from 185.220.101.6:21642 conn63456637: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
2021-06-24T01:34:08.018+0000 I COMMAND [conn63456637] dropDatabase config - starting
|
||||
2021-06-24T01:34:08.018+0000 I COMMAND [conn63456637] dropDatabase config - dropping 1 collections
|
||||
2021-06-24T01:34:08.018+0000 I COMMAND [conn63456637] dropDatabase config - dropping collection: config.transactions
|
||||
2021-06-24T01:34:08.020+0000 I STORAGE [conn63456637] dropCollection: config.transactions (no UUID) - renaming to drop-pending collection: config.system.drop.1624498448i1t-1.transactions with drop optime { ts: Timestamp(1624498448, 1), t: -1 }
|
||||
2021-06-24T01:34:08.029+0000 I REPL [replication-14545] Completing collection drop for config.system.drop.1624498448i1t-1.transactions with drop optime { ts: Timestamp(1624498448, 1), t: -1 } (notification optime: { ts: Timestamp(1624498448, 1), t: -1 })
|
||||
2021-06-24T01:34:08.030+0000 I STORAGE [replication-14545] Finishing collection drop for config.system.drop.1624498448i1t-1.transactions (no UUID).
|
||||
2021-06-24T01:34:08.030+0000 I COMMAND [conn63456637] dropDatabase config - successfully dropped 1 collections (most recent drop optime: { ts: Timestamp(1624498448, 1), t: -1 }) after 7ms. dropping database
|
||||
2021-06-24T01:34:08.032+0000 I REPL [replication-14546] Completing collection drop for config.system.drop.1624498448i1t-1.transactions with drop optime { ts: Timestamp(1624498448, 1), t: -1 } (notification optime: { ts: Timestamp(1624498448, 5), t: -1 })
|
||||
2021-06-24T01:34:08.041+0000 I COMMAND [conn63456637] dropDatabase config - finished
|
||||
2021-06-24T01:34:08.398+0000 I COMMAND [conn63456637] dropDatabase newsblur - starting
|
||||
2021-06-24T01:34:08.398+0000 I COMMAND [conn63456637] dropDatabase newsblur - dropping 37 collections
|
||||
|
||||
2021-06-24T01:33:45.531+0000 I NETWORK [listener] connection accepted from 171.25.193.78:26003 #63455699 (1189 connections now open)
|
||||
2021-06-24T01:33:45.635+0000 I NETWORK [conn63455699] received client metadata from 171.25.193.78:26003 conn63455699: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
2021-06-24T01:33:46.010+0000 I NETWORK [listener] connection accepted from 171.25.193.78:26557 #63455724 (1189 connections now open)
|
||||
2021-06-24T01:33:46.092+0000 I NETWORK [conn63455724] received client metadata from 171.25.193.78:26557 conn63455724: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
2021-06-24T01:33:46.500+0000 I NETWORK [conn63455724] end connection 171.25.193.78:26557 (1198 connections now open)
|
||||
2021-06-24T01:33:46.533+0000 I NETWORK [conn63455699] end connection 171.25.193.78:26003 (1200 connections now open)
|
||||
2021-06-24T01:34:06.533+0000 I NETWORK [listener] connection accepted from 185.220.101.6:10056 #63456621 (1266 connections now open)
|
||||
2021-06-24T01:34:06.627+0000 I NETWORK [conn63456621] received client metadata from 185.220.101.6:10056 conn63456621: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
2021-06-24T01:34:06.890+0000 I NETWORK [listener] connection accepted from 185.220.101.6:21642 #63456637 (1264 connections now open)
|
||||
2021-06-24T01:34:06.962+0000 I NETWORK [conn63456637] received client metadata from 185.220.101.6:21642 conn63456637: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
2021-06-24T01:34:08.018+0000 I COMMAND [conn63456637] dropDatabase config - starting
|
||||
2021-06-24T01:34:08.018+0000 I COMMAND [conn63456637] dropDatabase config - dropping 1 collections
|
||||
2021-06-24T01:34:08.018+0000 I COMMAND [conn63456637] dropDatabase config - dropping collection: config.transactions
|
||||
2021-06-24T01:34:08.020+0000 I STORAGE [conn63456637] dropCollection: config.transactions (no UUID) - renaming to drop-pending collection: config.system.drop.1624498448i1t-1.transactions with drop optime { ts: Timestamp(1624498448, 1), t: -1 }
|
||||
2021-06-24T01:34:08.029+0000 I REPL [replication-14545] Completing collection drop for config.system.drop.1624498448i1t-1.transactions with drop optime { ts: Timestamp(1624498448, 1), t: -1 } (notification optime: { ts: Timestamp(1624498448, 1), t: -1 })
|
||||
2021-06-24T01:34:08.030+0000 I STORAGE [replication-14545] Finishing collection drop for config.system.drop.1624498448i1t-1.transactions (no UUID).
|
||||
2021-06-24T01:34:08.030+0000 I COMMAND [conn63456637] dropDatabase config - successfully dropped 1 collections (most recent drop optime: { ts: Timestamp(1624498448, 1), t: -1 }) after 7ms. dropping database
|
||||
2021-06-24T01:34:08.032+0000 I REPL [replication-14546] Completing collection drop for config.system.drop.1624498448i1t-1.transactions with drop optime { ts: Timestamp(1624498448, 1), t: -1 } (notification optime: { ts: Timestamp(1624498448, 5), t: -1 })
|
||||
2021-06-24T01:34:08.041+0000 I COMMAND [conn63456637] dropDatabase config - finished
|
||||
2021-06-24T01:34:08.398+0000 I COMMAND [conn63456637] dropDatabase newsblur - starting
|
||||
2021-06-24T01:34:08.398+0000 I COMMAND [conn63456637] dropDatabase newsblur - dropping 37 collections
|
||||
<< SNIP: It goes on for a while... >>
|
||||
|
||||
<< SNIP: It goes on for a while... >>
|
||||
2021-06-24T01:35:18.840+0000 I COMMAND [conn63456637] dropDatabase newsblur - finished
|
||||
</code></pre></div></div>
|
||||
|
||||
2021-06-24T01:35:18.840+0000 I COMMAND [conn63456637] dropDatabase newsblur - finished
|
||||
The above is a lot, but the important bit of information to take from it is that by using a reductive filter, capturing everything that doesn't match a known IP, I was able to find the two connections that were made a few seconds apart. Both connections from these unknown IPs occured only moments before the database-wide deletion. By following the connection ID, it became easy to see the hacker come into the server only to delete it seconds later.
|
||||
|
||||
What you see above...
|
||||
|
||||
When I visited the IP address of the [two](http://185.220.101.6/) [connections](http://171.25.193.78/) above, I found a Tor exit router:
|
||||
Interestingly, when I visited the IP address of the [two](http://185.220.101.6/) [connections](http://171.25.193.78/) above, I found a Tor exit router:
|
||||
|
||||
<img src="/assets/hack-tor.png">
|
||||
|
||||
This means that it is virtually impossible to track down who is responsible due to the anonymity preserving quality of Tor exit routers. [Tor exit nodes have poor reputations](https://blog.cloudflare.com/the-trouble-with-tor/) due to the havoc they wreak. Site owners are split on whether to block Tor entirely, but some see the value of allowing anonymous traffic to hit their servers. In NewsBlur's case, because NewsBlur is a home of free speech, allowing users in countries with censored news outlets to bypass restrictions and get access to the world at wide, the continuing risk of supporting anonymous Internet traffic is worth the cost.
|
||||
|
||||
### 3. What will happen to ensure this doesn't happen again?
|
||||
|
||||
VPC all the way.
|
||||
Of course, being in support of free speech and providing enhanced ways to access speech comes at a cost. So for NewsBlur to continue serving traffic to all of its worldwide readers, several changes have to be made.
|
||||
|
||||
The first change is the one that, ironically, we were in the process of moving to. A VPC, a virtual private cloud, keeps critical servers only accessible from others servers in a private network. But in moving to a private network, I need to migrate all of the data off of the publicly accessible machines. And this was the first step in that process.
|
||||
|
||||
The second change is to use database user authentication on all of the databases. We had been relying on the firewall to provide protection against threats, but when the firewall silently failed, we were left exposed. Now who's to say that this would have been caught if the firewall failed but authentication was in place. I suspect the password needs to be long enough to not be brute-forced, because eventually, knowing that an open but password protected DB is there, it could very possibly end up on a list.
|
||||
|
||||
Lastly, a change needs to be made as to which database users have permission to drop the database. Most database users only need read and write privileges. The ideal would be a localhost-only user being allowed to perform potentially destructive actions. If a rogue database user starts deleting stories, it would get noticed a whole lot faster than a database being dropped all at once.
|
||||
|
||||
But each of these is only one piece of a defense strategy. [As this well-attended Hacker News thread from the day of the hack made clear](https://news.ycombinator.com/item?id=27613217), a proper defense strategy can never rely on only one move. And for NewsBlur that move was a allowlist-only firewall that worked perfectly up until it didn't.
|
||||
|
||||
As usually, the real heros are backups. Regularly well tested backups are a necessary component to any web service. And with that, I'll prepare to [launch the big NewsBlur redesign later this week](https://beta.newsblur.com).
|
||||
|
|
|
|||
|
|
@ -161,6 +161,8 @@ code {
|
|||
pre {
|
||||
padding: 8px 12px;
|
||||
overflow-x: auto;
|
||||
white-space: pre-wrap;
|
||||
word-wrap: break-word;
|
||||
|
||||
> code {
|
||||
border: 0;
|
||||
|
|
|
|||
|
|
@ -6,21 +6,21 @@
|
|||
<link rel="shortcut icon" href="https://newsblur.com/media/img/favicon.ico" type="image/png" />
|
||||
<link rel="icon" href="https://newsblur.com/media/img/favicon_32.png" sizes="32x32"/>
|
||||
<link rel="icon" href="https://newsblur.com/media/img/favicon_64.png" sizes="64x64"/><!-- Begin Jekyll SEO tag v2.7.1 -->
|
||||
<title>Story of a Hacking | The NewsBlur Blog</title>
|
||||
<title>How a Docker footgun led to a vandal deleting NewsBlur’s MongoDB database | The NewsBlur Blog</title>
|
||||
<meta name="generator" content="Jekyll v4.2.0" />
|
||||
<meta property="og:title" content="Story of a Hacking" />
|
||||
<meta property="og:title" content="How a Docker footgun led to a vandal deleting NewsBlur’s MongoDB database" />
|
||||
<meta property="og:locale" content="en_US" />
|
||||
<meta name="description" content="I’d like to answer a few questions about what happened here." />
|
||||
<meta property="og:description" content="I’d like to answer a few questions about what happened here." />
|
||||
<meta name="description" content="I’m in the process of moving everything on NewsBlur over to Docker containers in prep for a big redesign launching next week. It’s been a great year of maintenance and I’ve enjoyed the fruits of Ansible + Docker for NewsBlur’s 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into a new book on how to tame the machines once they’re smarter than us when I received a strange NewsBlur error on my phone." />
|
||||
<meta property="og:description" content="I’m in the process of moving everything on NewsBlur over to Docker containers in prep for a big redesign launching next week. It’s been a great year of maintenance and I’ve enjoyed the fruits of Ansible + Docker for NewsBlur’s 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into a new book on how to tame the machines once they’re smarter than us when I received a strange NewsBlur error on my phone." />
|
||||
<link rel="canonical" href="https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/" />
|
||||
<meta property="og:url" content="https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/" />
|
||||
<meta property="og:site_name" content="The NewsBlur Blog" />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="article:published_time" content="2021-06-25T00:00:00-04:00" />
|
||||
<meta name="twitter:card" content="summary" />
|
||||
<meta property="twitter:title" content="Story of a Hacking" />
|
||||
<meta property="twitter:title" content="How a Docker footgun led to a vandal deleting NewsBlur’s MongoDB database" />
|
||||
<script type="application/ld+json">
|
||||
{"description":"I’d like to answer a few questions about what happened here.","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/"},"publisher":{"@type":"Organization","logo":{"@type":"ImageObject","url":"https://blog2.newsblur.com/assets/newsblur_logo_512.png"}},"url":"https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/","headline":"Story of a Hacking","dateModified":"2021-06-25T00:00:00-04:00","datePublished":"2021-06-25T00:00:00-04:00","@type":"BlogPosting","@context":"https://schema.org"}</script>
|
||||
{"description":"I’m in the process of moving everything on NewsBlur over to Docker containers in prep for a big redesign launching next week. It’s been a great year of maintenance and I’ve enjoyed the fruits of Ansible + Docker for NewsBlur’s 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into a new book on how to tame the machines once they’re smarter than us when I received a strange NewsBlur error on my phone.","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/"},"publisher":{"@type":"Organization","logo":{"@type":"ImageObject","url":"https://blog2.newsblur.com/assets/newsblur_logo_512.png"}},"url":"https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/","headline":"How a Docker footgun led to a vandal deleting NewsBlur’s MongoDB database","dateModified":"2021-06-25T00:00:00-04:00","datePublished":"2021-06-25T00:00:00-04:00","@type":"BlogPosting","@context":"https://schema.org"}</script>
|
||||
<!-- End Jekyll SEO tag -->
|
||||
<link rel="stylesheet" href="/assets/main.css">
|
||||
<link rel="stylesheet" type="text/css" href="https://cloud.typography.com/6565292/711824/css/fonts.css" />
|
||||
|
|
@ -63,14 +63,45 @@
|
|||
<article class="post h-entry" itemscope itemtype="http://schema.org/BlogPosting">
|
||||
|
||||
<header class="post-header">
|
||||
<h1 class="post-title p-name" itemprop="name headline">Story of a Hacking</h1>
|
||||
<h1 class="post-title p-name" itemprop="name headline">How a Docker footgun led to a vandal deleting NewsBlur's MongoDB database</h1>
|
||||
<p class="post-meta">
|
||||
<time class="dt-published" datetime="2021-06-25T00:00:00-04:00" itemprop="datePublished">Jun 25, 2021
|
||||
</time></p>
|
||||
</header>
|
||||
|
||||
<div class="post-content e-content" itemprop="articleBody">
|
||||
<p>I’d like to answer a few questions about what happened here.</p>
|
||||
<p>I’m in the process of moving everything on NewsBlur over to Docker containers in prep for a <a href="https://beta.newsblur.com">big redesign launching next week</a>. It’s been a great year of maintenance and I’ve enjoyed the fruits of Ansible + Docker for NewsBlur’s 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into <a href="https://en.wikipedia.org/wiki/Human_Compatible">a new book on how to tame the machines once they’re smarter than us</a> when I received a strange NewsBlur error on my phone.</p>
|
||||
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"query killed during yield: renamed collection 'newsblur.feed_icons' to 'newsblur.system.drop.1624498448i220t-1.feed_icons'"
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>There are honestly no sets of words in that error message that I ever want to see again. What is the word <code class="language-plaintext highlighter-rouge">drop</code> doing in that error message? Better go find out.</p>
|
||||
|
||||
<p>Logging into the MongoDB machine to check out what state the DB is in and I come across the following…</p>
|
||||
|
||||
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">nbset</span><span class="p">:</span><span class="nx">PRIMARY</span><span class="o">></span> <span class="nx">show</span> <span class="nx">dbs</span>
|
||||
<span class="nx">READ__ME_TO_RECOVER_YOUR_DATA</span> <span class="mf">0.000</span><span class="nx">GB</span>
|
||||
<span class="nx">newsblur</span> <span class="mf">0.718</span><span class="nx">GB</span>
|
||||
|
||||
<span class="nx">nbset</span><span class="p">:</span><span class="nx">PRIMARY</span><span class="o">></span> <span class="nx">use</span> <span class="nx">READ__ME_TO_RECOVER_YOUR_DATA</span>
|
||||
<span class="nx">switched</span> <span class="nx">to</span> <span class="nx">db</span> <span class="nx">READ__ME_TO_RECOVER_YOUR_DATA</span>
|
||||
|
||||
<span class="nx">nbset</span><span class="p">:</span><span class="nx">PRIMARY</span><span class="o">></span> <span class="nx">db</span><span class="p">.</span><span class="nx">README</span><span class="p">.</span><span class="nx">find</span><span class="p">()</span>
|
||||
<span class="p">{</span>
|
||||
<span class="dl">"</span><span class="s2">_id</span><span class="dl">"</span> <span class="p">:</span> <span class="nx">ObjectId</span><span class="p">(</span><span class="dl">"</span><span class="s2">60d3e112ac48d82047aab95d</span><span class="dl">"</span><span class="p">),</span>
|
||||
<span class="dl">"</span><span class="s2">content</span><span class="dl">"</span> <span class="p">:</span> <span class="dl">"</span><span class="s2">All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump.</span><span class="dl">"</span>
|
||||
<span class="p">}</span></code></pre></figure>
|
||||
|
||||
<p>Two thoughts immediately occured:</p>
|
||||
|
||||
<ol>
|
||||
<li>Thank goodness I have some recently checked backups on hand</li>
|
||||
<li>No way they have that data without me noticing</li>
|
||||
</ol>
|
||||
|
||||
<p>Three and a half hours before this happened, I switched the MongoDB cluster over to the new servers. When I did that, I shut down the original primary in order to delete it in a few days when all was well. And thank goodness I did that as it came in handy a few hours later. Knowing this, I realized that the hacker could not have taken all that data in so little time.</p>
|
||||
|
||||
<p>With that in mind, I’d like to answer a few questions about what happened here.</p>
|
||||
|
||||
<ol>
|
||||
<li>Was any data leaked during the hack? How do you know?</li>
|
||||
|
|
@ -78,50 +109,46 @@
|
|||
<li>What will happen to ensure this doesn’t happen again?</li>
|
||||
</ol>
|
||||
|
||||
<p>Let’s start by talking about the importance of your data. As you may know, NewsBlur is open source and enjoys the added protection of having thousands of people looking at the codebase and dozens contributing back to it.</p>
|
||||
<p>Let’s start by talking about the most important question of all which is what happened to your data.</p>
|
||||
|
||||
<h3 id="1-was-any-data-leaked-during-the-hack-how-do-you-know">1. Was any data leaked during the hack? How do you know?</h3>
|
||||
|
||||
<p>I can definitively write that no data was leaked during the hack. I know this because of two different sets of logs showing that the automated attacker only issued deletion commands and did not transfer any data off of the MongoDB server.</p>
|
||||
|
||||
<p>This is what the day looks like. This 24 graph of bandwidth shows:</p>
|
||||
<p>Below is a snapshot of the bandwidth of the db-mongo1 machine over 24 hours:</p>
|
||||
|
||||
<p><img src="/assets/hack-timeline.png" style="border: 1px solid rgba(0,0,0,0.1);" /></p>
|
||||
|
||||
<p>You can imagine the stress I experienced in the forty minutes between 9:35p, when the hack began, and 10:15p, when the fresh backup snapshot was identified and put into gear.</p>
|
||||
<p>You can imagine the stress I experienced in the forty minutes between 9:35p, when the hack began, and 10:15p, when the fresh backup snapshot was identified and put into gear. Let’s breakdown each moment:</p>
|
||||
|
||||
<ol>
|
||||
<li><strong>6:10p</strong>:</li>
|
||||
<li><strong>9:35p</strong>:</li>
|
||||
<li><strong>10:15p</strong>:</li>
|
||||
<li><strong>3:00a</strong>:</li>
|
||||
<li><strong>4:30a</strong>:</li>
|
||||
<li><strong>6:10p</strong>: The new db-mongo1 server was put into rotation as the new MongoDB primary server. This machine was the first of the new, soon-to-be private cloud.</li>
|
||||
<li><strong>9:35p</strong>: Three hours later an automated hacking attempt opened a connection to the db-mongo1 server and immediately dropped the database. Downtime ensued.</li>
|
||||
<li><strong>10:15p</strong>: Before the former primary server could be placed into rotation, a snapshot of the server was made to ensure the backup would not delete itself upon reconnection. This cost a few hours of downtime, but saved nearly 18 hours of a day’s data by not forcing me to go into the daily backup archive.</li>
|
||||
<li><strong>3:00a</strong>: Snapshot completes, replication from original primary server to new db-mongo1 begins. What you see in the next hour and a half is what the transfer of the DB looks like in terms of bandwidth.</li>
|
||||
<li><strong>4:30a</strong>: Replication, which is inbound from the old primary server, completes, and now replication begins outbound on the new secondaries. NewsBlur is now back up.</li>
|
||||
</ol>
|
||||
|
||||
<p>The most important bit of information the above chart shows us is what a full database transfer looks like in terms of bandwidth. From 6p to 9:30p, the amount of data was the expected amount from a working primary server with multiple secondaries syncing to it. At 3a, you’ll see an enormous amount of data transfered.</p>
|
||||
|
||||
<p>This tells us that the hacker was an automated digital vandal rather than a concerted hacking attempt. And if we were to pay the ransom, it wouldn’t do anything because the vandals don’t have the data and have nothing to release.</p>
|
||||
|
||||
<p>While the server was being snapshot, I used that time to figure out how the hacker got in.</p>
|
||||
|
||||
<h3 id="2-how-did-newsblurs-mongodb-server-get-hacked">2. How did NewsBlur’s MongoDB server get hacked?</h3>
|
||||
|
||||
<p>It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun.</p>
|
||||
<p>Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn’t work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was “active”, doing a <code class="language-plaintext highlighter-rouge">sudo iptables -L | grep 27017</code> showed that MongoDB was open the world.</p>
|
||||
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nbset:PRIMARY> show dbs
|
||||
READ__ME_TO_RECOVER_YOUR_DATA 0.000GB
|
||||
admin 0.000GB
|
||||
local 16.471GB
|
||||
newsblur 0.718GB
|
||||
<p>To be honest, I’m a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlur’s MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored. And the Docker-made hole was immediately patched.</p>
|
||||
|
||||
nbset:PRIMARY> use READ__ME_TO_RECOVER_YOUR_DATA
|
||||
switched to db READ__ME_TO_RECOVER_YOUR_DATA
|
||||
<p>It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security. Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.</p>
|
||||
|
||||
nbset:PRIMARY> show collections
|
||||
README
|
||||
system.profile
|
||||
<p><img src="/assets/ornament-pill.png" style="display: block; margin: 0 auto;width: 100px;" /></p>
|
||||
|
||||
nbset:PRIMARY> db.README.find()
|
||||
{ "_id" : ObjectId("60d3e112ac48d82047aab95d"), "content" : "All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump." }
|
||||
</code></pre></div></div>
|
||||
<p>The second reason we know that no data was taken comes from looking through the MongoDB access logs. With these rich and verbose logging sources we can invoke a pretty neat command to find everybody who is not one of the 100 known NewsBlur machines that has accessed MongoDB.</p>
|
||||
|
||||
<p>Looking at the MongoDB access logs, we can invoke a pretty neat command to find everybody who is not one of the 100 known NewsBlur machines that has accessed MongoDB.</p>
|
||||
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat /var/log/mongodb/mongod.log | egrep -v "159.65.XX.XX|161.89.XX.XX|<< SNIP: A hundred more servers >>"
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight" style="max-height: 200px;"><code>
|
||||
$ cat /var/log/mongodb/mongod.log | egrep -v "159.65.XX.XX|161.89.XX.XX|<< SNIP: A hundred more servers >>"
|
||||
|
||||
2021-06-24T01:33:45.531+0000 I NETWORK [listener] connection accepted from 171.25.193.78:26003 #63455699 (1189 connections now open)
|
||||
2021-06-24T01:33:45.635+0000 I NETWORK [conn63455699] received client metadata from 171.25.193.78:26003 conn63455699: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
|
|
@ -150,15 +177,27 @@ nbset:PRIMARY> db.README.find()
|
|||
2021-06-24T01:35:18.840+0000 I COMMAND [conn63456637] dropDatabase newsblur - finished
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>What you see above…</p>
|
||||
<p>The above is a lot, but the important bit of information to take from it is that by using a reductive filter, capturing everything that doesn’t match a known IP, I was able to find the two connections that were made a few seconds apart. Both connections from these unknown IPs occured only moments before the database-wide deletion. By following the connection ID, it became easy to see the hacker come into the server only to delete it seconds later.</p>
|
||||
|
||||
<p>When I visited the IP address of the <a href="http://185.220.101.6/">two</a> <a href="http://171.25.193.78/">connections</a> above, I found a Tor exit router:</p>
|
||||
<p>Interestingly, when I visited the IP address of the <a href="http://185.220.101.6/">two</a> <a href="http://171.25.193.78/">connections</a> above, I found a Tor exit router:</p>
|
||||
|
||||
<p><img src="/assets/hack-tor.png" /></p>
|
||||
|
||||
<p>This means that it is virtually impossible to track down who is responsible due to the anonymity preserving quality of Tor exit routers. <a href="https://blog.cloudflare.com/the-trouble-with-tor/">Tor exit nodes have poor reputations</a> due to the havoc they wreak. Site owners are split on whether to block Tor entirely, but some see the value of allowing anonymous traffic to hit their servers. In NewsBlur’s case, because NewsBlur is a home of free speech, allowing users in countries with censored news outlets to bypass restrictions and get access to the world at wide, the continuing risk of supporting anonymous Internet traffic is worth the cost.</p>
|
||||
|
||||
<h3 id="3-what-will-happen-to-ensure-this-doesnt-happen-again">3. What will happen to ensure this doesn’t happen again?</h3>
|
||||
|
||||
<p>VPC all the way.</p>
|
||||
<p>Of course, being in support of free speech and providing enhanced ways to access speech comes at a cost. So for NewsBlur to continue serving traffic to all of its worldwide readers, several changes have to be made.</p>
|
||||
|
||||
<p>The first change is the one that, ironically, we were in the process of moving to. A VPC, a virtual private cloud, keeps critical servers only accessible from others servers in a private network. But in moving to a private network, I need to migrate all of the data off of the publicly accessible machines. And this was the first step in that process.</p>
|
||||
|
||||
<p>The second change is to use database user authentication on all of the databases. We had been relying on the firewall to provide protection against threats, but when the firewall silently failed, we were left exposed. Now who’s to say that this would have been caught if the firewall failed but authentication was in place. I suspect the password needs to be long enough to not be brute-forced, because eventually, knowing that an open but password protected DB is there, it could very possibly end up on a list.</p>
|
||||
|
||||
<p>Lastly, a change needs to be made as to which database users have permission to drop the database. Most database users only need read and write privileges. The ideal would be a localhost-only user being allowed to perform potentially destructive actions. If a rogue database user starts deleting stories, it would get noticed a whole lot faster than a database being dropped all at once.</p>
|
||||
|
||||
<p>But each of these is only one piece of a defense strategy. <a href="https://news.ycombinator.com/item?id=27613217">As this well-attended Hacker News thread from the day of the hack made clear</a>, a proper defense strategy can never rely on only one move. And for NewsBlur that move was a allowlist-only firewall that worked perfectly up until it didn’t.</p>
|
||||
|
||||
<p>As usually, the real heros are backups. Regularly well tested backups are a necessary component to any web service. And with that, I’ll prepare to <a href="https://beta.newsblur.com">launch the big NewsBlur redesign later this week</a>.</p>
|
||||
|
||||
</div><a class="u-url" href="/2021/06/25/story-of-a-hacking/" hidden></a>
|
||||
</article>
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@ pre, code { font-size: 15px; border: 1px solid #e5e8e4; border-radius: 3px; back
|
|||
|
||||
code { padding: 1px 5px; }
|
||||
|
||||
pre { padding: 8px 12px; overflow-x: auto; }
|
||||
pre { padding: 8px 12px; overflow-x: auto; white-space: pre-wrap; word-wrap: break-word; }
|
||||
|
||||
pre > code { border: 0; padding-right: 0; padding-left: 0; }
|
||||
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
BIN
blog/_site/assets/ornament-pill.png
Normal file
BIN
blog/_site/assets/ornament-pill.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.7 KiB |
|
|
@ -1,6 +1,37 @@
|
|||
<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.2.0">Jekyll</generator><link href="https://blog2.newsblur.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://blog2.newsblur.com/" rel="alternate" type="text/html" /><updated>2021-06-25T18:27:17-04:00</updated><id>https://blog2.newsblur.com/feed.xml</id><title type="html">The NewsBlur Blog</title><subtitle>NewsBlur is a personal news reader that brings people together to talk about the world.
|
||||
<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.2.0">Jekyll</generator><link href="https://blog2.newsblur.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://blog2.newsblur.com/" rel="alternate" type="text/html" /><updated>2021-06-27T21:03:53-04:00</updated><id>https://blog2.newsblur.com/feed.xml</id><title type="html">The NewsBlur Blog</title><subtitle>NewsBlur is a personal news reader that brings people together to talk about the world.
|
||||
A new sound of an old instrument.
|
||||
</subtitle><entry><title type="html">Story of a Hacking</title><link href="https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/" rel="alternate" type="text/html" title="Story of a Hacking" /><published>2021-06-25T00:00:00-04:00</published><updated>2021-06-25T00:00:00-04:00</updated><id>https://blog2.newsblur.com/2021/06/25/story-of-a-hacking</id><content type="html" xml:base="https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/"><p>I’d like to answer a few questions about what happened here.</p>
|
||||
</subtitle><entry><title type="html">How a Docker footgun led to a vandal deleting NewsBlur’s MongoDB database</title><link href="https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/" rel="alternate" type="text/html" title="How a Docker footgun led to a vandal deleting NewsBlur’s MongoDB database" /><published>2021-06-25T00:00:00-04:00</published><updated>2021-06-25T00:00:00-04:00</updated><id>https://blog2.newsblur.com/2021/06/25/story-of-a-hacking</id><content type="html" xml:base="https://blog2.newsblur.com/2021/06/25/story-of-a-hacking/"><p>I’m in the process of moving everything on NewsBlur over to Docker containers in prep for a <a href="https://beta.newsblur.com">big redesign launching next week</a>. It’s been a great year of maintenance and I’ve enjoyed the fruits of Ansible + Docker for NewsBlur’s 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into <a href="https://en.wikipedia.org/wiki/Human_Compatible">a new book on how to tame the machines once they’re smarter than us</a> when I received a strange NewsBlur error on my phone.</p>
|
||||
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"query killed during yield: renamed collection 'newsblur.feed_icons' to 'newsblur.system.drop.1624498448i220t-1.feed_icons'"
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>There are honestly no sets of words in that error message that I ever want to see again. What is the word <code class="language-plaintext highlighter-rouge">drop</code> doing in that error message? Better go find out.</p>
|
||||
|
||||
<p>Logging into the MongoDB machine to check out what state the DB is in and I come across the following…</p>
|
||||
|
||||
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">nbset</span><span class="p">:</span><span class="nx">PRIMARY</span><span class="o">&gt;</span> <span class="nx">show</span> <span class="nx">dbs</span>
|
||||
<span class="nx">READ__ME_TO_RECOVER_YOUR_DATA</span> <span class="mf">0.000</span><span class="nx">GB</span>
|
||||
<span class="nx">newsblur</span> <span class="mf">0.718</span><span class="nx">GB</span>
|
||||
|
||||
<span class="nx">nbset</span><span class="p">:</span><span class="nx">PRIMARY</span><span class="o">&gt;</span> <span class="nx">use</span> <span class="nx">READ__ME_TO_RECOVER_YOUR_DATA</span>
|
||||
<span class="nx">switched</span> <span class="nx">to</span> <span class="nx">db</span> <span class="nx">READ__ME_TO_RECOVER_YOUR_DATA</span>
|
||||
|
||||
<span class="nx">nbset</span><span class="p">:</span><span class="nx">PRIMARY</span><span class="o">&gt;</span> <span class="nx">db</span><span class="p">.</span><span class="nx">README</span><span class="p">.</span><span class="nx">find</span><span class="p">()</span>
|
||||
<span class="p">{</span>
|
||||
<span class="dl">"</span><span class="s2">_id</span><span class="dl">"</span> <span class="p">:</span> <span class="nx">ObjectId</span><span class="p">(</span><span class="dl">"</span><span class="s2">60d3e112ac48d82047aab95d</span><span class="dl">"</span><span class="p">),</span>
|
||||
<span class="dl">"</span><span class="s2">content</span><span class="dl">"</span> <span class="p">:</span> <span class="dl">"</span><span class="s2">All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump.</span><span class="dl">"</span>
|
||||
<span class="p">}</span></code></pre></figure>
|
||||
|
||||
<p>Two thoughts immediately occured:</p>
|
||||
|
||||
<ol>
|
||||
<li>Thank goodness I have some recently checked backups on hand</li>
|
||||
<li>No way they have that data without me noticing</li>
|
||||
</ol>
|
||||
|
||||
<p>Three and a half hours before this happened, I switched the MongoDB cluster over to the new servers. When I did that, I shut down the original primary in order to delete it in a few days when all was well. And thank goodness I did that as it came in handy a few hours later. Knowing this, I realized that the hacker could not have taken all that data in so little time.</p>
|
||||
|
||||
<p>With that in mind, I’d like to answer a few questions about what happened here.</p>
|
||||
|
||||
<ol>
|
||||
<li>Was any data leaked during the hack? How do you know?</li>
|
||||
|
|
@ -8,50 +39,46 @@ A new sound of an old instrument.
|
|||
<li>What will happen to ensure this doesn’t happen again?</li>
|
||||
</ol>
|
||||
|
||||
<p>Let’s start by talking about the importance of your data. As you may know, NewsBlur is open source and enjoys the added protection of having thousands of people looking at the codebase and dozens contributing back to it.</p>
|
||||
<p>Let’s start by talking about the most important question of all which is what happened to your data.</p>
|
||||
|
||||
<h3 id="1-was-any-data-leaked-during-the-hack-how-do-you-know">1. Was any data leaked during the hack? How do you know?</h3>
|
||||
|
||||
<p>I can definitively write that no data was leaked during the hack. I know this because of two different sets of logs showing that the automated attacker only issued deletion commands and did not transfer any data off of the MongoDB server.</p>
|
||||
|
||||
<p>This is what the day looks like. This 24 graph of bandwidth shows:</p>
|
||||
<p>Below is a snapshot of the bandwidth of the db-mongo1 machine over 24 hours:</p>
|
||||
|
||||
<p><img src="/assets/hack-timeline.png" style="border: 1px solid rgba(0,0,0,0.1);" /></p>
|
||||
|
||||
<p>You can imagine the stress I experienced in the forty minutes between 9:35p, when the hack began, and 10:15p, when the fresh backup snapshot was identified and put into gear.</p>
|
||||
<p>You can imagine the stress I experienced in the forty minutes between 9:35p, when the hack began, and 10:15p, when the fresh backup snapshot was identified and put into gear. Let’s breakdown each moment:</p>
|
||||
|
||||
<ol>
|
||||
<li><strong>6:10p</strong>:</li>
|
||||
<li><strong>9:35p</strong>:</li>
|
||||
<li><strong>10:15p</strong>:</li>
|
||||
<li><strong>3:00a</strong>:</li>
|
||||
<li><strong>4:30a</strong>:</li>
|
||||
<li><strong>6:10p</strong>: The new db-mongo1 server was put into rotation as the new MongoDB primary server. This machine was the first of the new, soon-to-be private cloud.</li>
|
||||
<li><strong>9:35p</strong>: Three hours later an automated hacking attempt opened a connection to the db-mongo1 server and immediately dropped the database. Downtime ensued.</li>
|
||||
<li><strong>10:15p</strong>: Before the former primary server could be placed into rotation, a snapshot of the server was made to ensure the backup would not delete itself upon reconnection. This cost a few hours of downtime, but saved nearly 18 hours of a day’s data by not forcing me to go into the daily backup archive.</li>
|
||||
<li><strong>3:00a</strong>: Snapshot completes, replication from original primary server to new db-mongo1 begins. What you see in the next hour and a half is what the transfer of the DB looks like in terms of bandwidth.</li>
|
||||
<li><strong>4:30a</strong>: Replication, which is inbound from the old primary server, completes, and now replication begins outbound on the new secondaries. NewsBlur is now back up.</li>
|
||||
</ol>
|
||||
|
||||
<p>The most important bit of information the above chart shows us is what a full database transfer looks like in terms of bandwidth. From 6p to 9:30p, the amount of data was the expected amount from a working primary server with multiple secondaries syncing to it. At 3a, you’ll see an enormous amount of data transfered.</p>
|
||||
|
||||
<p>This tells us that the hacker was an automated digital vandal rather than a concerted hacking attempt. And if we were to pay the ransom, it wouldn’t do anything because the vandals don’t have the data and have nothing to release.</p>
|
||||
|
||||
<p>While the server was being snapshot, I used that time to figure out how the hacker got in.</p>
|
||||
|
||||
<h3 id="2-how-did-newsblurs-mongodb-server-get-hacked">2. How did NewsBlur’s MongoDB server get hacked?</h3>
|
||||
|
||||
<p>It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun.</p>
|
||||
<p>Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn’t work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was “active”, doing a <code class="language-plaintext highlighter-rouge">sudo iptables -L | grep 27017</code> showed that MongoDB was open the world.</p>
|
||||
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nbset:PRIMARY&gt; show dbs
|
||||
READ__ME_TO_RECOVER_YOUR_DATA 0.000GB
|
||||
admin 0.000GB
|
||||
local 16.471GB
|
||||
newsblur 0.718GB
|
||||
<p>To be honest, I’m a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlur’s MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored. And the Docker-made hole was immediately patched.</p>
|
||||
|
||||
nbset:PRIMARY&gt; use READ__ME_TO_RECOVER_YOUR_DATA
|
||||
switched to db READ__ME_TO_RECOVER_YOUR_DATA
|
||||
<p>It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security. Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.</p>
|
||||
|
||||
nbset:PRIMARY&gt; show collections
|
||||
README
|
||||
system.profile
|
||||
<p><img src="/assets/ornament-pill.png" style="display: block; margin: 0 auto;width: 100px;" /></p>
|
||||
|
||||
nbset:PRIMARY&gt; db.README.find()
|
||||
{ "_id" : ObjectId("60d3e112ac48d82047aab95d"), "content" : "All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump." }
|
||||
</code></pre></div></div>
|
||||
<p>The second reason we know that no data was taken comes from looking through the MongoDB access logs. With these rich and verbose logging sources we can invoke a pretty neat command to find everybody who is not one of the 100 known NewsBlur machines that has accessed MongoDB.</p>
|
||||
|
||||
<p>Looking at the MongoDB access logs, we can invoke a pretty neat command to find everybody who is not one of the 100 known NewsBlur machines that has accessed MongoDB.</p>
|
||||
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat /var/log/mongodb/mongod.log | egrep -v "159.65.XX.XX|161.89.XX.XX|&lt;&lt; SNIP: A hundred more servers &gt;&gt;"
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight" style="max-height: 200px;"><code>
|
||||
$ cat /var/log/mongodb/mongod.log | egrep -v "159.65.XX.XX|161.89.XX.XX|&lt;&lt; SNIP: A hundred more servers &gt;&gt;"
|
||||
|
||||
2021-06-24T01:33:45.531+0000 I NETWORK [listener] connection accepted from 171.25.193.78:26003 #63455699 (1189 connections now open)
|
||||
2021-06-24T01:33:45.635+0000 I NETWORK [conn63455699] received client metadata from 171.25.193.78:26003 conn63455699: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
|
|
@ -80,15 +107,27 @@ nbset:PRIMARY&gt; db.README.find()
|
|||
2021-06-24T01:35:18.840+0000 I COMMAND [conn63456637] dropDatabase newsblur - finished
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>What you see above…</p>
|
||||
<p>The above is a lot, but the important bit of information to take from it is that by using a reductive filter, capturing everything that doesn’t match a known IP, I was able to find the two connections that were made a few seconds apart. Both connections from these unknown IPs occured only moments before the database-wide deletion. By following the connection ID, it became easy to see the hacker come into the server only to delete it seconds later.</p>
|
||||
|
||||
<p>When I visited the IP address of the <a href="http://185.220.101.6/">two</a> <a href="http://171.25.193.78/">connections</a> above, I found a Tor exit router:</p>
|
||||
<p>Interestingly, when I visited the IP address of the <a href="http://185.220.101.6/">two</a> <a href="http://171.25.193.78/">connections</a> above, I found a Tor exit router:</p>
|
||||
|
||||
<p><img src="/assets/hack-tor.png" /></p>
|
||||
|
||||
<p>This means that it is virtually impossible to track down who is responsible due to the anonymity preserving quality of Tor exit routers. <a href="https://blog.cloudflare.com/the-trouble-with-tor/">Tor exit nodes have poor reputations</a> due to the havoc they wreak. Site owners are split on whether to block Tor entirely, but some see the value of allowing anonymous traffic to hit their servers. In NewsBlur’s case, because NewsBlur is a home of free speech, allowing users in countries with censored news outlets to bypass restrictions and get access to the world at wide, the continuing risk of supporting anonymous Internet traffic is worth the cost.</p>
|
||||
|
||||
<h3 id="3-what-will-happen-to-ensure-this-doesnt-happen-again">3. What will happen to ensure this doesn’t happen again?</h3>
|
||||
|
||||
<p>VPC all the way.</p></content><author><name></name></author><category term="backend" /><summary type="html">I’d like to answer a few questions about what happened here.</summary></entry><entry><title type="html">Android app update: premium subscriptions, saved searches, in-app browser, auto-dark mode</title><link href="https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved/" rel="alternate" type="text/html" title="Android app update: premium subscriptions, saved searches, in-app browser, auto-dark mode" /><published>2020-11-03T07:41:03-05:00</published><updated>2020-11-03T07:41:03-05:00</updated><id>https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved</id><content type="html" xml:base="https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved/"><p>For a point release this one sure is big. The Android app has been upgraded to include a bunch of features found on the web.</p>
|
||||
<p>Of course, being in support of free speech and providing enhanced ways to access speech comes at a cost. So for NewsBlur to continue serving traffic to all of its worldwide readers, several changes have to be made.</p>
|
||||
|
||||
<p>The first change is the one that, ironically, we were in the process of moving to. A VPC, a virtual private cloud, keeps critical servers only accessible from others servers in a private network. But in moving to a private network, I need to migrate all of the data off of the publicly accessible machines. And this was the first step in that process.</p>
|
||||
|
||||
<p>The second change is to use database user authentication on all of the databases. We had been relying on the firewall to provide protection against threats, but when the firewall silently failed, we were left exposed. Now who’s to say that this would have been caught if the firewall failed but authentication was in place. I suspect the password needs to be long enough to not be brute-forced, because eventually, knowing that an open but password protected DB is there, it could very possibly end up on a list.</p>
|
||||
|
||||
<p>Lastly, a change needs to be made as to which database users have permission to drop the database. Most database users only need read and write privileges. The ideal would be a localhost-only user being allowed to perform potentially destructive actions. If a rogue database user starts deleting stories, it would get noticed a whole lot faster than a database being dropped all at once.</p>
|
||||
|
||||
<p>But each of these is only one piece of a defense strategy. <a href="https://news.ycombinator.com/item?id=27613217">As this well-attended Hacker News thread from the day of the hack made clear</a>, a proper defense strategy can never rely on only one move. And for NewsBlur that move was a allowlist-only firewall that worked perfectly up until it didn’t.</p>
|
||||
|
||||
<p>As usually, the real heros are backups. Regularly well tested backups are a necessary component to any web service. And with that, I’ll prepare to <a href="https://beta.newsblur.com">launch the big NewsBlur redesign later this week</a>.</p></content><author><name></name></author><category term="backend" /><summary type="html">I’m in the process of moving everything on NewsBlur over to Docker containers in prep for a big redesign launching next week. It’s been a great year of maintenance and I’ve enjoyed the fruits of Ansible + Docker for NewsBlur’s 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into a new book on how to tame the machines once they’re smarter than us when I received a strange NewsBlur error on my phone.</summary></entry><entry><title type="html">Android app update: premium subscriptions, saved searches, in-app browser, auto-dark mode</title><link href="https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved/" rel="alternate" type="text/html" title="Android app update: premium subscriptions, saved searches, in-app browser, auto-dark mode" /><published>2020-11-03T07:41:03-05:00</published><updated>2020-11-03T07:41:03-05:00</updated><id>https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved</id><content type="html" xml:base="https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved/"><p>For a point release this one sure is big. The Android app has been upgraded to include a bunch of features found on the web.</p>
|
||||
|
||||
<p>For one, premium subscriptions can now be purchased in the Android app itself. Reading by folder, saved story tags, searching and saved searches are all premium features that you can unlock directly in the app.</p>
|
||||
|
||||
|
|
|
|||
|
|
@ -63,11 +63,42 @@
|
|||
<ul class="post-list"><li><span class="post-meta">Jun 25, 2021</span>
|
||||
<h3>
|
||||
<a class="post-link" href="/2021/06/25/story-of-a-hacking/">
|
||||
Story of a Hacking
|
||||
How a Docker footgun led to a vandal deleting NewsBlur's MongoDB database
|
||||
</a>
|
||||
</h3>
|
||||
<div class="post-content e-content" itemprop="articleBody">
|
||||
<p>I’d like to answer a few questions about what happened here.</p>
|
||||
<p>I’m in the process of moving everything on NewsBlur over to Docker containers in prep for a <a href="https://beta.newsblur.com">big redesign launching next week</a>. It’s been a great year of maintenance and I’ve enjoyed the fruits of Ansible + Docker for NewsBlur’s 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into <a href="https://en.wikipedia.org/wiki/Human_Compatible">a new book on how to tame the machines once they’re smarter than us</a> when I received a strange NewsBlur error on my phone.</p>
|
||||
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"query killed during yield: renamed collection 'newsblur.feed_icons' to 'newsblur.system.drop.1624498448i220t-1.feed_icons'"
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>There are honestly no sets of words in that error message that I ever want to see again. What is the word <code class="language-plaintext highlighter-rouge">drop</code> doing in that error message? Better go find out.</p>
|
||||
|
||||
<p>Logging into the MongoDB machine to check out what state the DB is in and I come across the following…</p>
|
||||
|
||||
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">nbset</span><span class="p">:</span><span class="nx">PRIMARY</span><span class="o">></span> <span class="nx">show</span> <span class="nx">dbs</span>
|
||||
<span class="nx">READ__ME_TO_RECOVER_YOUR_DATA</span> <span class="mf">0.000</span><span class="nx">GB</span>
|
||||
<span class="nx">newsblur</span> <span class="mf">0.718</span><span class="nx">GB</span>
|
||||
|
||||
<span class="nx">nbset</span><span class="p">:</span><span class="nx">PRIMARY</span><span class="o">></span> <span class="nx">use</span> <span class="nx">READ__ME_TO_RECOVER_YOUR_DATA</span>
|
||||
<span class="nx">switched</span> <span class="nx">to</span> <span class="nx">db</span> <span class="nx">READ__ME_TO_RECOVER_YOUR_DATA</span>
|
||||
|
||||
<span class="nx">nbset</span><span class="p">:</span><span class="nx">PRIMARY</span><span class="o">></span> <span class="nx">db</span><span class="p">.</span><span class="nx">README</span><span class="p">.</span><span class="nx">find</span><span class="p">()</span>
|
||||
<span class="p">{</span>
|
||||
<span class="dl">"</span><span class="s2">_id</span><span class="dl">"</span> <span class="p">:</span> <span class="nx">ObjectId</span><span class="p">(</span><span class="dl">"</span><span class="s2">60d3e112ac48d82047aab95d</span><span class="dl">"</span><span class="p">),</span>
|
||||
<span class="dl">"</span><span class="s2">content</span><span class="dl">"</span> <span class="p">:</span> <span class="dl">"</span><span class="s2">All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump.</span><span class="dl">"</span>
|
||||
<span class="p">}</span></code></pre></figure>
|
||||
|
||||
<p>Two thoughts immediately occured:</p>
|
||||
|
||||
<ol>
|
||||
<li>Thank goodness I have some recently checked backups on hand</li>
|
||||
<li>No way they have that data without me noticing</li>
|
||||
</ol>
|
||||
|
||||
<p>Three and a half hours before this happened, I switched the MongoDB cluster over to the new servers. When I did that, I shut down the original primary in order to delete it in a few days when all was well. And thank goodness I did that as it came in handy a few hours later. Knowing this, I realized that the hacker could not have taken all that data in so little time.</p>
|
||||
|
||||
<p>With that in mind, I’d like to answer a few questions about what happened here.</p>
|
||||
|
||||
<ol>
|
||||
<li>Was any data leaked during the hack? How do you know?</li>
|
||||
|
|
@ -75,50 +106,46 @@
|
|||
<li>What will happen to ensure this doesn’t happen again?</li>
|
||||
</ol>
|
||||
|
||||
<p>Let’s start by talking about the importance of your data. As you may know, NewsBlur is open source and enjoys the added protection of having thousands of people looking at the codebase and dozens contributing back to it.</p>
|
||||
<p>Let’s start by talking about the most important question of all which is what happened to your data.</p>
|
||||
|
||||
<h3 id="1-was-any-data-leaked-during-the-hack-how-do-you-know">1. Was any data leaked during the hack? How do you know?</h3>
|
||||
|
||||
<p>I can definitively write that no data was leaked during the hack. I know this because of two different sets of logs showing that the automated attacker only issued deletion commands and did not transfer any data off of the MongoDB server.</p>
|
||||
|
||||
<p>This is what the day looks like. This 24 graph of bandwidth shows:</p>
|
||||
<p>Below is a snapshot of the bandwidth of the db-mongo1 machine over 24 hours:</p>
|
||||
|
||||
<p><img src="/assets/hack-timeline.png" style="border: 1px solid rgba(0,0,0,0.1);" /></p>
|
||||
|
||||
<p>You can imagine the stress I experienced in the forty minutes between 9:35p, when the hack began, and 10:15p, when the fresh backup snapshot was identified and put into gear.</p>
|
||||
<p>You can imagine the stress I experienced in the forty minutes between 9:35p, when the hack began, and 10:15p, when the fresh backup snapshot was identified and put into gear. Let’s breakdown each moment:</p>
|
||||
|
||||
<ol>
|
||||
<li><strong>6:10p</strong>:</li>
|
||||
<li><strong>9:35p</strong>:</li>
|
||||
<li><strong>10:15p</strong>:</li>
|
||||
<li><strong>3:00a</strong>:</li>
|
||||
<li><strong>4:30a</strong>:</li>
|
||||
<li><strong>6:10p</strong>: The new db-mongo1 server was put into rotation as the new MongoDB primary server. This machine was the first of the new, soon-to-be private cloud.</li>
|
||||
<li><strong>9:35p</strong>: Three hours later an automated hacking attempt opened a connection to the db-mongo1 server and immediately dropped the database. Downtime ensued.</li>
|
||||
<li><strong>10:15p</strong>: Before the former primary server could be placed into rotation, a snapshot of the server was made to ensure the backup would not delete itself upon reconnection. This cost a few hours of downtime, but saved nearly 18 hours of a day’s data by not forcing me to go into the daily backup archive.</li>
|
||||
<li><strong>3:00a</strong>: Snapshot completes, replication from original primary server to new db-mongo1 begins. What you see in the next hour and a half is what the transfer of the DB looks like in terms of bandwidth.</li>
|
||||
<li><strong>4:30a</strong>: Replication, which is inbound from the old primary server, completes, and now replication begins outbound on the new secondaries. NewsBlur is now back up.</li>
|
||||
</ol>
|
||||
|
||||
<p>The most important bit of information the above chart shows us is what a full database transfer looks like in terms of bandwidth. From 6p to 9:30p, the amount of data was the expected amount from a working primary server with multiple secondaries syncing to it. At 3a, you’ll see an enormous amount of data transfered.</p>
|
||||
|
||||
<p>This tells us that the hacker was an automated digital vandal rather than a concerted hacking attempt. And if we were to pay the ransom, it wouldn’t do anything because the vandals don’t have the data and have nothing to release.</p>
|
||||
|
||||
<p>While the server was being snapshot, I used that time to figure out how the hacker got in.</p>
|
||||
|
||||
<h3 id="2-how-did-newsblurs-mongodb-server-get-hacked">2. How did NewsBlur’s MongoDB server get hacked?</h3>
|
||||
|
||||
<p>It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun.</p>
|
||||
<p>Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn’t work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was “active”, doing a <code class="language-plaintext highlighter-rouge">sudo iptables -L | grep 27017</code> showed that MongoDB was open the world.</p>
|
||||
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nbset:PRIMARY> show dbs
|
||||
READ__ME_TO_RECOVER_YOUR_DATA 0.000GB
|
||||
admin 0.000GB
|
||||
local 16.471GB
|
||||
newsblur 0.718GB
|
||||
<p>To be honest, I’m a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlur’s MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored. And the Docker-made hole was immediately patched.</p>
|
||||
|
||||
nbset:PRIMARY> use READ__ME_TO_RECOVER_YOUR_DATA
|
||||
switched to db READ__ME_TO_RECOVER_YOUR_DATA
|
||||
<p>It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security. Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.</p>
|
||||
|
||||
nbset:PRIMARY> show collections
|
||||
README
|
||||
system.profile
|
||||
<p><img src="/assets/ornament-pill.png" style="display: block; margin: 0 auto;width: 100px;" /></p>
|
||||
|
||||
nbset:PRIMARY> db.README.find()
|
||||
{ "_id" : ObjectId("60d3e112ac48d82047aab95d"), "content" : "All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump." }
|
||||
</code></pre></div></div>
|
||||
<p>The second reason we know that no data was taken comes from looking through the MongoDB access logs. With these rich and verbose logging sources we can invoke a pretty neat command to find everybody who is not one of the 100 known NewsBlur machines that has accessed MongoDB.</p>
|
||||
|
||||
<p>Looking at the MongoDB access logs, we can invoke a pretty neat command to find everybody who is not one of the 100 known NewsBlur machines that has accessed MongoDB.</p>
|
||||
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat /var/log/mongodb/mongod.log | egrep -v "159.65.XX.XX|161.89.XX.XX|<< SNIP: A hundred more servers >>"
|
||||
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight" style="max-height: 200px;"><code>
|
||||
$ cat /var/log/mongodb/mongod.log | egrep -v "159.65.XX.XX|161.89.XX.XX|<< SNIP: A hundred more servers >>"
|
||||
|
||||
2021-06-24T01:33:45.531+0000 I NETWORK [listener] connection accepted from 171.25.193.78:26003 #63455699 (1189 connections now open)
|
||||
2021-06-24T01:33:45.635+0000 I NETWORK [conn63455699] received client metadata from 171.25.193.78:26003 conn63455699: { driver: { name: "PyMongo", version: "3.11.4" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-74-generic" }, platform: "CPython 3.8.5.final.0" }
|
||||
|
|
@ -147,15 +174,27 @@ nbset:PRIMARY> db.README.find()
|
|||
2021-06-24T01:35:18.840+0000 I COMMAND [conn63456637] dropDatabase newsblur - finished
|
||||
</code></pre></div></div>
|
||||
|
||||
<p>What you see above…</p>
|
||||
<p>The above is a lot, but the important bit of information to take from it is that by using a reductive filter, capturing everything that doesn’t match a known IP, I was able to find the two connections that were made a few seconds apart. Both connections from these unknown IPs occured only moments before the database-wide deletion. By following the connection ID, it became easy to see the hacker come into the server only to delete it seconds later.</p>
|
||||
|
||||
<p>When I visited the IP address of the <a href="http://185.220.101.6/">two</a> <a href="http://171.25.193.78/">connections</a> above, I found a Tor exit router:</p>
|
||||
<p>Interestingly, when I visited the IP address of the <a href="http://185.220.101.6/">two</a> <a href="http://171.25.193.78/">connections</a> above, I found a Tor exit router:</p>
|
||||
|
||||
<p><img src="/assets/hack-tor.png" /></p>
|
||||
|
||||
<p>This means that it is virtually impossible to track down who is responsible due to the anonymity preserving quality of Tor exit routers. <a href="https://blog.cloudflare.com/the-trouble-with-tor/">Tor exit nodes have poor reputations</a> due to the havoc they wreak. Site owners are split on whether to block Tor entirely, but some see the value of allowing anonymous traffic to hit their servers. In NewsBlur’s case, because NewsBlur is a home of free speech, allowing users in countries with censored news outlets to bypass restrictions and get access to the world at wide, the continuing risk of supporting anonymous Internet traffic is worth the cost.</p>
|
||||
|
||||
<h3 id="3-what-will-happen-to-ensure-this-doesnt-happen-again">3. What will happen to ensure this doesn’t happen again?</h3>
|
||||
|
||||
<p>VPC all the way.</p>
|
||||
<p>Of course, being in support of free speech and providing enhanced ways to access speech comes at a cost. So for NewsBlur to continue serving traffic to all of its worldwide readers, several changes have to be made.</p>
|
||||
|
||||
<p>The first change is the one that, ironically, we were in the process of moving to. A VPC, a virtual private cloud, keeps critical servers only accessible from others servers in a private network. But in moving to a private network, I need to migrate all of the data off of the publicly accessible machines. And this was the first step in that process.</p>
|
||||
|
||||
<p>The second change is to use database user authentication on all of the databases. We had been relying on the firewall to provide protection against threats, but when the firewall silently failed, we were left exposed. Now who’s to say that this would have been caught if the firewall failed but authentication was in place. I suspect the password needs to be long enough to not be brute-forced, because eventually, knowing that an open but password protected DB is there, it could very possibly end up on a list.</p>
|
||||
|
||||
<p>Lastly, a change needs to be made as to which database users have permission to drop the database. Most database users only need read and write privileges. The ideal would be a localhost-only user being allowed to perform potentially destructive actions. If a rogue database user starts deleting stories, it would get noticed a whole lot faster than a database being dropped all at once.</p>
|
||||
|
||||
<p>But each of these is only one piece of a defense strategy. <a href="https://news.ycombinator.com/item?id=27613217">As this well-attended Hacker News thread from the day of the hack made clear</a>, a proper defense strategy can never rely on only one move. And for NewsBlur that move was a allowlist-only firewall that worked perfectly up until it didn’t.</p>
|
||||
|
||||
<p>As usually, the real heros are backups. Regularly well tested backups are a necessary component to any web service. And with that, I’ll prepare to <a href="https://beta.newsblur.com">launch the big NewsBlur redesign later this week</a>.</p>
|
||||
|
||||
</div>
|
||||
</li><li><span class="post-meta">Nov 3, 2020</span>
|
||||
|
|
|
|||
BIN
blog/assets/ornament-pill.png
Normal file
BIN
blog/assets/ornament-pill.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.7 KiB |
Loading…
Add table
Reference in a new issue