Updating blog post, pushing public on blog.newsblur.com.

This commit is contained in:
Samuel Clay 2021-06-28 18:29:57 -04:00
parent bca23d3850
commit 569df99676
5 changed files with 27 additions and 19 deletions

View file

@ -4,13 +4,13 @@ title: How a Docker footgun led to a vandal deleting NewsBlur's MongoDB database
tags: ['backend']
---
*tl;dr: A vandal deleted NewsBlur's MongoDB during a migration. No data was stolen or lost.*
*tl;dr: A vandal deleted NewsBlur's MongoDB database during a migration. No data was stolen or lost.*
I'm in the process of moving everything on NewsBlur over to Docker containers in prep for a [big redesign launching next week](https://beta.newsblur.com). It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into [a new book on how to tame the machines once they're smarter than us](https://en.wikipedia.org/wiki/Human_Compatible) when I received a strange NewsBlur error on my phone.
"query killed during yield: renamed collection 'newsblur.feed_icons' to 'newsblur.system.drop.1624498448i220t-1.feed_icons'"
There are honestly no sets of words in that error message that I ever want to see again. What is the word `drop` doing in that error message? Better go find out.
There is honestly no set of words in that error message that I ever want to see again. What is `drop` doing in that error message? Better go find out.
Logging into the MongoDB machine to check out what state the DB is in and I come across the following...
@ -64,13 +64,15 @@ The most important bit of information the above chart shows us is what a full da
This tells us that the hacker was an automated digital vandal rather than a concerted hacking attempt. And if we were to pay the ransom, it wouldn't do anything because the vandals don't have the data and have nothing to release.
We can also reason that the vandal was not able to access any files that were on the server outside of MongoDB due to using a recent version of MongoDB. Unless the attacker had access to a 0-day, it is highly unlikely they were able to break out of MongoDB's server connection.
While the server was being snapshot, I used that time to figure out how the hacker got in.
### 2. How did NewsBlur's MongoDB server get hacked?
Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn't work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was "active", doing a `sudo iptables -L | grep 27017` showed that MongoDB was open the world.
To be honest, I'm a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlur's MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored. And the Docker-made hole was immediately patched.
To be honest, I'm a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlur's MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored and the Docker-made hole was patched.
It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security. Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.

View file

@ -10,8 +10,8 @@
<meta name="generator" content="Jekyll v4.2.0" />
<meta property="og:title" content="How a Docker footgun led to a vandal deleting NewsBlurs MongoDB database" />
<meta property="og:locale" content="en_US" />
<meta name="description" content="tl;dr: A vandal deleted NewsBlurs MongoDB during a migration. No data was stolen or lost." />
<meta property="og:description" content="tl;dr: A vandal deleted NewsBlurs MongoDB during a migration. No data was stolen or lost." />
<meta name="description" content="tl;dr: A vandal deleted NewsBlurs MongoDB database during a migration. No data was stolen or lost." />
<meta property="og:description" content="tl;dr: A vandal deleted NewsBlurs MongoDB database during a migration. No data was stolen or lost." />
<link rel="canonical" href="https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/" />
<meta property="og:url" content="https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/" />
<meta property="og:site_name" content="The NewsBlur Blog" />
@ -20,7 +20,7 @@
<meta name="twitter:card" content="summary" />
<meta property="twitter:title" content="How a Docker footgun led to a vandal deleting NewsBlurs MongoDB database" />
<script type="application/ld+json">
{"@type":"BlogPosting","publisher":{"@type":"Organization","logo":{"@type":"ImageObject","url":"https://blog2.newsblur.com/assets/newsblur_logo_512.png"}},"headline":"How a Docker footgun led to a vandal deleting NewsBlurs MongoDB database","dateModified":"2021-06-28T00:00:00-04:00","datePublished":"2021-06-28T00:00:00-04:00","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/"},"description":"tl;dr: A vandal deleted NewsBlurs MongoDB during a migration. No data was stolen or lost.","url":"https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/","@context":"https://schema.org"}</script>
{"@type":"BlogPosting","publisher":{"@type":"Organization","logo":{"@type":"ImageObject","url":"https://blog2.newsblur.com/assets/newsblur_logo_512.png"}},"headline":"How a Docker footgun led to a vandal deleting NewsBlurs MongoDB database","dateModified":"2021-06-28T00:00:00-04:00","datePublished":"2021-06-28T00:00:00-04:00","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/"},"description":"tl;dr: A vandal deleted NewsBlurs MongoDB database during a migration. No data was stolen or lost.","url":"https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/","@context":"https://schema.org"}</script>
<!-- End Jekyll SEO tag -->
<link rel="stylesheet" href="/assets/main.css">
<link rel="stylesheet" type="text/css" href="https://cloud.typography.com/6565292/711824/css/fonts.css" />
@ -70,14 +70,14 @@
</header>
<div class="post-content e-content" itemprop="articleBody">
<p><em>tl;dr: A vandal deleted NewsBlurs MongoDB during a migration. No data was stolen or lost.</em></p>
<p><em>tl;dr: A vandal deleted NewsBlurs MongoDB database during a migration. No data was stolen or lost.</em></p>
<p>Im in the process of moving everything on NewsBlur over to Docker containers in prep for a <a href="https://beta.newsblur.com">big redesign launching next week</a>. Its been a great year of maintenance and Ive enjoyed the fruits of Ansible + Docker for NewsBlurs 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into <a href="https://en.wikipedia.org/wiki/Human_Compatible">a new book on how to tame the machines once theyre smarter than us</a> when I received a strange NewsBlur error on my phone.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"query killed during yield: renamed collection 'newsblur.feed_icons' to 'newsblur.system.drop.1624498448i220t-1.feed_icons'"
</code></pre></div></div>
<p>There are honestly no sets of words in that error message that I ever want to see again. What is the word <code class="language-plaintext highlighter-rouge">drop</code> doing in that error message? Better go find out.</p>
<p>There is honestly no set of words in that error message that I ever want to see again. What is <code class="language-plaintext highlighter-rouge">drop</code> doing in that error message? Better go find out.</p>
<p>Logging into the MongoDB machine to check out what state the DB is in and I come across the following…</p>
@ -135,13 +135,15 @@
<p>This tells us that the hacker was an automated digital vandal rather than a concerted hacking attempt. And if we were to pay the ransom, it wouldnt do anything because the vandals dont have the data and have nothing to release.</p>
<p>We can also reason that the vandal was not able to access any files that were on the server outside of MongoDB due to using a recent version of MongoDB. Unless the attacker had access to a 0-day, it is highly unlikely they were able to break out of MongoDBs server connection.</p>
<p>While the server was being snapshot, I used that time to figure out how the hacker got in.</p>
<h3 id="2-how-did-newsblurs-mongodb-server-get-hacked">2. How did NewsBlurs MongoDB server get hacked?</h3>
<p>Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didnt work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was “active”, doing a <code class="language-plaintext highlighter-rouge">sudo iptables -L | grep 27017</code> showed that MongoDB was open the world.</p>
<p>To be honest, Im a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlurs MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored. And the Docker-made hole was immediately patched.</p>
<p>To be honest, Im a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlurs MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored and the Docker-made hole was patched.</p>
<p>It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security. Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.</p>

View file

@ -1,13 +1,13 @@
<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.2.0">Jekyll</generator><link href="https://blog2.newsblur.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://blog2.newsblur.com/" rel="alternate" type="text/html" /><updated>2021-06-28T18:12:20-04:00</updated><id>https://blog2.newsblur.com/feed.xml</id><title type="html">The NewsBlur Blog</title><subtitle>NewsBlur is a personal news reader that brings people together to talk about the world.
<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.2.0">Jekyll</generator><link href="https://blog2.newsblur.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://blog2.newsblur.com/" rel="alternate" type="text/html" /><updated>2021-06-28T18:29:37-04:00</updated><id>https://blog2.newsblur.com/feed.xml</id><title type="html">The NewsBlur Blog</title><subtitle>NewsBlur is a personal news reader that brings people together to talk about the world.
A new sound of an old instrument.
</subtitle><entry><title type="html">How a Docker footgun led to a vandal deleting NewsBlurs MongoDB database</title><link href="https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/" rel="alternate" type="text/html" title="How a Docker footgun led to a vandal deleting NewsBlurs MongoDB database" /><published>2021-06-28T00:00:00-04:00</published><updated>2021-06-28T00:00:00-04:00</updated><id>https://blog2.newsblur.com/2021/06/28/story-of-a-hacking</id><content type="html" xml:base="https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/">&lt;p&gt;&lt;em&gt;tl;dr: A vandal deleted NewsBlurs MongoDB during a migration. No data was stolen or lost.&lt;/em&gt;&lt;/p&gt;
</subtitle><entry><title type="html">How a Docker footgun led to a vandal deleting NewsBlurs MongoDB database</title><link href="https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/" rel="alternate" type="text/html" title="How a Docker footgun led to a vandal deleting NewsBlurs MongoDB database" /><published>2021-06-28T00:00:00-04:00</published><updated>2021-06-28T00:00:00-04:00</updated><id>https://blog2.newsblur.com/2021/06/28/story-of-a-hacking</id><content type="html" xml:base="https://blog2.newsblur.com/2021/06/28/story-of-a-hacking/">&lt;p&gt;&lt;em&gt;tl;dr: A vandal deleted NewsBlurs MongoDB database during a migration. No data was stolen or lost.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Im in the process of moving everything on NewsBlur over to Docker containers in prep for a &lt;a href=&quot;https://beta.newsblur.com&quot;&gt;big redesign launching next week&lt;/a&gt;. Its been a great year of maintenance and Ive enjoyed the fruits of Ansible + Docker for NewsBlurs 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into &lt;a href=&quot;https://en.wikipedia.org/wiki/Human_Compatible&quot;&gt;a new book on how to tame the machines once theyre smarter than us&lt;/a&gt; when I received a strange NewsBlur error on my phone.&lt;/p&gt;
&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&quot;query killed during yield: renamed collection 'newsblur.feed_icons' to 'newsblur.system.drop.1624498448i220t-1.feed_icons'&quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;There are honestly no sets of words in that error message that I ever want to see again. What is the word &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;drop&lt;/code&gt; doing in that error message? Better go find out.&lt;/p&gt;
&lt;p&gt;There is honestly no set of words in that error message that I ever want to see again. What is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;drop&lt;/code&gt; doing in that error message? Better go find out.&lt;/p&gt;
&lt;p&gt;Logging into the MongoDB machine to check out what state the DB is in and I come across the following…&lt;/p&gt;
@ -65,13 +65,15 @@ A new sound of an old instrument.
&lt;p&gt;This tells us that the hacker was an automated digital vandal rather than a concerted hacking attempt. And if we were to pay the ransom, it wouldnt do anything because the vandals dont have the data and have nothing to release.&lt;/p&gt;
&lt;p&gt;We can also reason that the vandal was not able to access any files that were on the server outside of MongoDB due to using a recent version of MongoDB. Unless the attacker had access to a 0-day, it is highly unlikely they were able to break out of MongoDBs server connection.&lt;/p&gt;
&lt;p&gt;While the server was being snapshot, I used that time to figure out how the hacker got in.&lt;/p&gt;
&lt;h3 id=&quot;2-how-did-newsblurs-mongodb-server-get-hacked&quot;&gt;2. How did NewsBlurs MongoDB server get hacked?&lt;/h3&gt;
&lt;p&gt;Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didnt work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was “active”, doing a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;sudo iptables -L | grep 27017&lt;/code&gt; showed that MongoDB was open the world.&lt;/p&gt;
&lt;p&gt;To be honest, Im a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlurs MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored. And the Docker-made hole was immediately patched.&lt;/p&gt;
&lt;p&gt;To be honest, Im a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlurs MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored and the Docker-made hole was patched.&lt;/p&gt;
&lt;p&gt;It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security. Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.&lt;/p&gt;
@ -129,7 +131,7 @@ $ cat /var/log/mongodb/mongod.log | egrep -v &quot;159.65.XX.XX|161.89.XX.XX|&am
&lt;p&gt;But each of these is only one piece of a defense strategy. &lt;a href=&quot;https://news.ycombinator.com/item?id=27613217&quot;&gt;As this well-attended Hacker News thread from the day of the hack made clear&lt;/a&gt;, a proper defense strategy can never rely on only one well-setup layer. And for NewsBlur that layer was a allowlist-only firewall that worked perfectly up until it didnt.&lt;/p&gt;
&lt;p&gt;As usually, the real heros are backups. Regular, well-tested backups are a necessary component to any web service. And with that, Ill prepare to &lt;a href=&quot;https://beta.newsblur.com&quot;&gt;launch the big NewsBlur redesign later this week&lt;/a&gt;.&lt;/p&gt;</content><author><name></name></author><category term="backend" /><summary type="html">tl;dr: A vandal deleted NewsBlurs MongoDB during a migration. No data was stolen or lost.</summary></entry><entry><title type="html">Android app update: premium subscriptions, saved searches, in-app browser, auto-dark mode</title><link href="https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved/" rel="alternate" type="text/html" title="Android app update: premium subscriptions, saved searches, in-app browser, auto-dark mode" /><published>2020-11-03T07:41:03-05:00</published><updated>2020-11-03T07:41:03-05:00</updated><id>https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved</id><content type="html" xml:base="https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved/">&lt;p&gt;For a point release this one sure is big. The Android app has been upgraded to include a bunch of features found on the web.&lt;/p&gt;
&lt;p&gt;As usually, the real heros are backups. Regular, well-tested backups are a necessary component to any web service. And with that, Ill prepare to &lt;a href=&quot;https://beta.newsblur.com&quot;&gt;launch the big NewsBlur redesign later this week&lt;/a&gt;.&lt;/p&gt;</content><author><name></name></author><category term="backend" /><summary type="html">tl;dr: A vandal deleted NewsBlurs MongoDB database during a migration. No data was stolen or lost.</summary></entry><entry><title type="html">Android app update: premium subscriptions, saved searches, in-app browser, auto-dark mode</title><link href="https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved/" rel="alternate" type="text/html" title="Android app update: premium subscriptions, saved searches, in-app browser, auto-dark mode" /><published>2020-11-03T07:41:03-05:00</published><updated>2020-11-03T07:41:03-05:00</updated><id>https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved</id><content type="html" xml:base="https://blog2.newsblur.com/2020/11/03/android-app-update-premium-subscriptions-saved/">&lt;p&gt;For a point release this one sure is big. The Android app has been upgraded to include a bunch of features found on the web.&lt;/p&gt;
&lt;p&gt;For one, premium subscriptions can now be purchased in the Android app itself. Reading by folder, saved story tags, searching and saved searches are all premium features that you can unlock directly in the app.&lt;/p&gt;

View file

@ -67,14 +67,14 @@
</a>
</h3>
<div class="post-content e-content" itemprop="articleBody">
<p><em>tl;dr: A vandal deleted NewsBlurs MongoDB during a migration. No data was stolen or lost.</em></p>
<p><em>tl;dr: A vandal deleted NewsBlurs MongoDB database during a migration. No data was stolen or lost.</em></p>
<p>Im in the process of moving everything on NewsBlur over to Docker containers in prep for a <a href="https://beta.newsblur.com">big redesign launching next week</a>. Its been a great year of maintenance and Ive enjoyed the fruits of Ansible + Docker for NewsBlurs 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). The day was wrapping up and I settled into <a href="https://en.wikipedia.org/wiki/Human_Compatible">a new book on how to tame the machines once theyre smarter than us</a> when I received a strange NewsBlur error on my phone.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"query killed during yield: renamed collection 'newsblur.feed_icons' to 'newsblur.system.drop.1624498448i220t-1.feed_icons'"
</code></pre></div></div>
<p>There are honestly no sets of words in that error message that I ever want to see again. What is the word <code class="language-plaintext highlighter-rouge">drop</code> doing in that error message? Better go find out.</p>
<p>There is honestly no set of words in that error message that I ever want to see again. What is <code class="language-plaintext highlighter-rouge">drop</code> doing in that error message? Better go find out.</p>
<p>Logging into the MongoDB machine to check out what state the DB is in and I come across the following…</p>
@ -132,13 +132,15 @@
<p>This tells us that the hacker was an automated digital vandal rather than a concerted hacking attempt. And if we were to pay the ransom, it wouldnt do anything because the vandals dont have the data and have nothing to release.</p>
<p>We can also reason that the vandal was not able to access any files that were on the server outside of MongoDB due to using a recent version of MongoDB. Unless the attacker had access to a 0-day, it is highly unlikely they were able to break out of MongoDBs server connection.</p>
<p>While the server was being snapshot, I used that time to figure out how the hacker got in.</p>
<h3 id="2-how-did-newsblurs-mongodb-server-get-hacked">2. How did NewsBlurs MongoDB server get hacked?</h3>
<p>Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didnt work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was “active”, doing a <code class="language-plaintext highlighter-rouge">sudo iptables -L | grep 27017</code> showed that MongoDB was open the world.</p>
<p>To be honest, Im a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlurs MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored. And the Docker-made hole was immediately patched.</p>
<p>To be honest, Im a bit surprised it took over 3 hours from when I flipped the switch to when a hacker/vandal dropped NewsBlurs MongoDB collections and pretended to ransom about 250GB of data. This is the work of an automated hack and one that I was prepared for. NewsBlur was back online a few hours later once the backups were restored and the Docker-made hole was patched.</p>
<p>It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security. Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.</p>

View file

@ -118,13 +118,13 @@ server {
listen 81;
{% endif %}
server_name blog2.newsblur.com;
server_name blog.newsblur.com blog2.newsblur.com;
set_real_ip_from 0.0.0.0/0;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
rewrite ^/rss$ https://blog2.newsblur.com/feed.xml permanent;
rewrite ^/rss$ https://blog.newsblur.com/feed.xml permanent;
root /srv/newsblur/blog/_site;
}