public-mirrors/NewsBlur
1
0
Fork 0
mirror of https://github.com/samuelclay/NewsBlur.git synced 2025-09-18 21:50:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update README.md and remove commented lines from docker-compose.yml

Browse source
This commit is contained in:
Jonathan Math 2020-11-03 11:26:04 +07:00
parent 80bae3608d
commit fd7643a078
2 changed files with 47 additions and 268 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

312
README.md
View file

@ -21,8 +21,7 @@
## Technologies
### Server-side
* [Python 2.7+](http://www.python.org): The language of choice.
* [Python 3.7+](http://www.python.org): The language of choice.
* [Django](http://www.djangoproject.com): Web framework written in Python, used
to serve all pages.
* [Celery](http://ask.github.com/celery) & [RabbitMQ](http://www.rabbitmq.com):
@ -46,192 +45,66 @@
[See the complete list](https://github.com/samuelclay/NewsBlur/tree/master/media/js).
## Installation Instructions
### Prerequisites
* Docker
* Docker-Compose
#### What you can safely ignore
## Installation Instructions
1. Run `make nb` to build Newsblur containers. This will set up all necessary databases, celery tasks, node applications,
flask database monitor, NGINX, and a Haproxy load balancer.
Not every program listed in the Prerequisites section is necessary to run NewsBlur.
2. Navigate to:
* `Elasticsearch` is the only module that requires Java. If you can live without searching for feeds or searching for stories, then you can ignore it and NewsBlur will just spit out that you don't have a search server in the logs.
* `Jammit` is for asset compression. Don't bother using it since the alternative is to just serve every js and css file in individual files without compression. Besides, nginx gzips those files automatically if you use the built-in nginx config. Just set `DEBUG_ASSETS = True` in your local_settings.py (which is also in local_settings.py.template).
* `numpy` and `scipy` are used for the colors used all over the site. Every site's favicon is analyzed for its dominant color, and that color is what gives every site its feel. You'll see it by every story all over. I'd recommend installing it, as you can just use prebuilt packages and don't have to install from source, which is possible but not trivial.
#### Relational Database (MySQL, PostgreSQL)
You will want to have your database set up before you begin installation. Fabric can install
both PostgreSQL and MongoDB for you, but only on Ubuntu. Mac OS X users will want to have
MySQL or PostgreSQL already installed. You can [download MySQL](http://dev.mysql.com/downloads/mysql/)
or [download PostgreSQL](http://www.postgresql.org/download/). Additionally,
if running as a development machine on Mac OS X, I would recommend using MySQL with
[Sequel Pro](http://www.sequelpro.com/) as a GUI.
If you are installing MySQL, you will also need the MySQLDB python library:
sudo easy_install mysql-python
#### Fabric
Both Mac OS X and Linux require [Fabric](http://docs.fabfile.org/) to be installed.
Many common tasks, such as installing dependencies, deploying servers, migrations,
and configurations are in `fabfile.py`.
sudo easy_install fabric
On recent installations of Mac OS X using XCode 4, you may run into issues around the
`ppc` architecture. To fix this, simply run:
sudo ln -s /Developer/Platforms/iPhoneOS.platform/Developer/usr/libexec/gcc/darwin/ppc \
/Developer/usr/libexec/gcc/darwin
sudo ln -s /Developer/Platforms/iPhoneOS.platform/Developer/usr/libexec/gcc/darwin/ppc \
/usr/libexec/gcc/darwin
Sym-linking the ppc architecture comes from this StackOverflow answer on
"[assembler for architecture ppc not installed on Mac OS](http://stackoverflow.com/questions/5256397/)".
#### MongoDB
On top of MySQL/PostgreSQL, NewsBlur uses MongoDB to store non-relational data. You will want to
[download MongoDB](http://www.mongodb.org/downloads). If you are on Ubuntu, the `setup_mongo` Fabric
command will automatically do this for you, but Mac OS X needs to have it installed manually.
#### Numpy and Scipy
Not the easiest to get installed. If you are running Mac OS X, you have a few options:
* Use the [Superpack by Chris Fonnesbeck](http://fonnesbeck.github.com/ScipySuperpack/)
* Use MacPorts: `sudo port install py26-numpy py26-scipy`
* Install from source (grueling): [http://www.scipy.org/Download](http://www.scipy.org/Download)
* Use a combination of pip, easy_install, and [homebrew](http://mxcl.github.com/homebrew/): `pip install numpy && brew install gfortran && easy_install scipy`
#### Jammit
You must have Java 7 installed to run Jammit.
* Install Java 7 on OS X by following directions from http://www.cc.gatech.edu/~simpkins/teaching/gatech/cs2340/guides/java7-macosx.html
* Also install the following gems:
`sudo gem install closure-compiler jsmin cssmin uglifier`
#### Other assorted packages
From inside the repository, run:
pip install -r requirements.txt
### Configure paths
In `fabfile.py` there are two paths that need to be configured.
* `env.paths.NEWSBLUR` is the relative path to the NewsBlur repository.
* `env.paths.VENDOR` is the relative path to where all downloaded code should go.
In `local_settings.py` there are a few paths that need to be configured. Configure
these after the installation below.
### Installing on Mac OS X
1. Using Mac OS X as a development environment, you can run all three servers (app, db, task)
on the same system. You should have [Fabric](http://docs.fabfile.org/) installed to run
the `fabfile.py`. You should also have MySQL/PostgreSQL and MongoDB already installed.
fab -R local setup_python
fab -R local setup_mongoengine
fab -R local setup_forked_mongoengine
fab -R local setup_repo_local_settings
fab -R local compress_assets
If any of the packages fail to install (`lxml`, for instance), look through `fabfile.py`
and check if there is a function that can be used to circumvent broken easy_install
processes. For example, lxml may need libxslt and libxml2 to be installed. This is
automated with the following Fabric command:
fab -R local setup_libxml_code
2. Configure MySQL/PostgreSQL by adding in a `newsblur` user and a `newsblur` database. Here's an example for MySQL:
mysql_install_db --verbose --user=`whoami` --basedir="$(brew --prefix mysql)" --datadir=/path/to/var/mysql --tmpdir=/tmp
mysql.server start
mysql -u root
> CREATE USER 'newsblur'@'localhost' IDENTIFIED BY '';
> GRANT ALL PRIVILEGES ON *.* TO 'newsblur'@'localhost' WITH GRANT OPTION;
> CREATE DATABASE newsblur;
> exit
Then load up the database with empty NewsBlur tables and bootstrap the database:
./manage.py syncdb --all --noinput
./manage.py migrate --fake
./manage.py migrate
./manage.py loaddata config/fixtures/bootstrap.json
3. Start mongodb (if not already running):
mongod run
4. Run the development server. At this point, all dependencies should be installed and no
additional configuration is needed. If you find that something is not working at this
point, please email the resulting output to Samuel Clay at
[samuel@newsblur.com](mailto:samuel@newsblur.com).
./manage.py runserver --settings=newsblur_web.settings.py
5. Navigate to:
http://localhost:8000/
https://nb.local.com
Create an account. At the end of the account creation process, you
will be redirected to https://localhost/profile/stripe_form. Hit
the back button a few times, and you will be inside the app.
### Installing on Linux / Ubuntu
If you are on Ubuntu, you can simply use [Fabric](http://docs.fabfile.org/) to install
NewsBlur and its many components. NewsBlur is designed to run on three separate servers:
an app server, a db server, and assorted task servers. To install everything on a single
machine, read through `fabfile.py` and setup all three servers (app, db, and task) without
repeating the `setup_common` steps.
## Contribution Instructions
### Finishing Installation
* Making Changes:
* To apply changes to the Python or JavaScript code, use the `make` command.
* To apply changes to the docker-compose.yml file, use the `make rebuild` command.
* To apply changes to the docker/haproxy/haproxy.conf file, node packages, or any new database migrations you will need to use the `make nb` command.
You must perform a few tasks to tie all of the various systems together.
* Adding Python packages:
Currently, the docker-compose.yml file uses the newsblur/newsblur_python3 image. It is built using the Dockerfile found in `docker/newsblur_base_image.Dockerfile`. Because of how the docker image is set up, you will need to create your own image and direct your docker-compose.yml file to use it. Please follow the following steps to do so.
1. First, copy local_settings.py and fill in your OAuth keys, S3 keys, database names
(if not `newsblur`), task server/broker address (RabbitMQ), and paths:
1. Add your new site-packages to config/requirements.txt.
2. Add the following lines of code to your docker-compose.yml file to replace anywhere where it says `image: newsblur/newsblur_python3`
cp local_settings.py.template local_settings.py
Edit local_settings.py to change any keys that you have.
<code>
build:
context: .
dockerfile: docker/newsblur_base_image.Dockerfile
</code>
2. Create the `newsblur` database in MySQL/PostgreSQL
3. Run the `make nb` command to rebuild your docker-compose containers
#### MySQL/PostgreSQL
./manage.py migrate
* Debugging Python
* To debug your code, drop `import pdb; pdb.set_trace()` into the Python code where you would like to start debugging
and run `make` and then `make debug`.
* Using Django shell within Docker
* Make sure your docker containers are up and run `make shell` to open
the Django shell within the newsblur_web container.
### Running unit and integration tests
NewsBlur comes complete with a test suite that tests the functionality of the rss_feeds,
reader, and feed importer. To run the test suite:
`make test`
#### App server
fab -R local setup_app
#### Database server
fab -R local setup_db
#### Task server
fab -R local setup_task
## Keeping NewsBlur Running
## Keeping NewsBlur Running **** is this still applicable?
These commands keep NewsBlur fresh and updated. While on a development server, these
commands do not need to be run more than once. However, you will probably want to run
the `refresh_feeds` command regularly so you have new stories to test with and read.
### Fetching feeds
### Fetching feeds **** is this still applicable?
If you just want to fetch feeds once, you can use the `refresh_feeds` management command:
@ -247,21 +120,21 @@ You can also fetch the feeds for a specific user:
You'll want to put this `refresh_feeds` command on a timer to keep your feeds up to date.
### Feedback
### Feedback **** is this still applicable?
To populate the feedback table on the homepage, use the `collect_feedback` management
command every few minutes:
./manage.py collect_feedback
### Statistics
### Statistics **** is this still applicable?
To populate the statistics graphs on the homepage, use the `collect_stats` management
command every few minutes:
./manage.py collect_stats
### Bootstrapping Search
### Bootstrapping Search **** is this still applicable?
Once you have an elasticsearch server running, you'll want to bootstrap it with feed and story indexes.
@ -269,107 +142,16 @@ Once you have an elasticsearch server running, you'll want to bootstrap it with
Stories will be indexed automatically.
If you need to move search servers and want to just delete everything in the search database, you need to reset the MUserSearch table.
If you need to move search servers and want to just delete everything in the search database, you need to reset the MUserSearch table. Run
`make shell`
>>> from apps.search.models import MUserSearch
>>> MUserSearch.remove_all()
### Running unit and integration tests
NewsBlur comes complete with a test suite that tests the functionality of the rss_feeds,
reader, and feed importer. To run the test suite:
./manage.py test --settings=newsblur_web.test-settings
## In Case of Downtime
You got the downtime message either through email or SMS. This is the order of operations for determining what's wrong.
0a. If downtime goes over 5 minutes, go to Twitter and say you're handling it. Be transparent about what it is,
NewsBlur's followers are largely technical. Also the 502 page points users to Twitter for status updates.
0b. Ensure you have `secrets-newsblur/configs/hosts` installed in your `/etc/hosts` so server hostnames
work.
1. Check www.newsblur.com to confirm it's down.
### If feeds aren't fetching:
check that the `tasked_feeds` queue is empty. You can drain it by running:
`make shell`
If you don't get a 502 page, then NewsBlur isn't even reachable and you just need to contact [the
hosting provider](https://cloudsupport.digitalocean.com/s/createticket) and yell at them.
2. Check which servers can't be reached on HAProxy stats page. Basic auth can be found in secrets/configs/haproxy.conf. Search the secrets repo for "gimmiestats".
Typically it'll be mongo, but any of the redis or postgres servers can be unreachable due to
acts of god. Otherwise, a frequent cause is lack of disk space. There are monitors on every DB
server watching for disk space, emailing me when they're running low, but it still happens.
3. Check [Sentry](https://app.getsentry.com/newsblur/app/) and see if the answer is at the top of the
list.
This will show if a database (redis, mongo, postgres) can't be found.
4. Check the various databases:
a. If Redis server (db_redis, db_redis_story, db_redis_pubsub) can't connect, redis is probably down.
SSH into the offending server (or just check both the `db_redis` and `db_redis_story` servers) and
check if `redis` is running. You can often `tail -f -n 100 /var/log/redis.log` to find out if
background saving was being SIG(TERM|INT)'ed. When redis goes down, it's always because it's
consuming too much memory. That shouldn't happen, so check the [munin
graphs](http://db_redis/munin/).
Boot it with `sudo /etc/init.d/redis start`.
b. If mongo (db_mongo) can't connect, mongo is probably down.
This is rare and usually signifies hardware failure. SSH into `db_mongo` and check logs with `tail
-f -n 100 /var/log/mongodb/mongodb.log`. Start mongo with `sudo /etc/init.d/mongodb start` then
promote the next largest mongodb server. You want to then promote one of the secondaries to
primary, kill the offending primary machine, and rebuild it (preferably at a higher size). I
recommend waiting a day to rebuild it so that you get a different machine. Don't forget to lodge a
support ticket with the hosting provider so they know to check the machine.
If it's the db_mongo_analytics machine, there is no backup nor secondaries of the data (because
it's ephemeral and used for, you guessed it, analytics). You can easily provision a new mongodb
server and point to that machine.
If mongo is out of space, which happens, the servers need to be re-synced every 2-3 months to
compress the data bloat. Simply `rm -fr /var/lib/mongodb/*` and re-start Mongo. It will re-sync.
If both secondaries are down, then the primary Mongo will go down. You'll need a secondary mongo
in the sync state at the very least before the primary will accept reads. It shouldn't take long to
get into that state, but you'll need a mongodb machine setup. You can immediately reuse the
non-working secondary if disk space is the only issue.
c. If postgresql (db_pgsql) can't connect, postgres is probably down.
This is the rarest of the rare and has in fact never happened. Machine failure. If you can salvage
the db data, move it to another machine. Worst case you have nightly backups in S3. The fabfile.py
has commands to assist in restoring from backup (the backup file just needs to be local).
4. Point to a new/different machine
a. Confirm the IP address of the new machine with `fab list_do`.
b. Change `secrets-newsbur/config/hosts` to reflect the new machine.
c. Copy the new `hosts` file to all machines with:
```
fab all setup_hosts
```
d. Changes should be instant, but you can also bounce every machine with:
```
fab web deploy
fab task celery
```
e. Monitor `utils/tlnb.py` and `utils/tlnbt.py` for lots of reading and feed fetching.
5. If feeds aren't fetching, check that the `tasked_feeds` queue is empty. You can drain it by running:
```
Feed.drain_task_feeds()
```

3
docker-compose.yml
View file

@ -56,9 +56,6 @@ services:
postgres:
container_name: postgres
image: postgres:9.6
#build:
# context: .
# dockerfile: docker/pg_docker/Dockerfile
environment:
- POSTGRES_USER=newsblur
- POSTGRES_PASSWORD=newsblur