Discourse in the Reclaim Cloud

What’s old is new again. In 2015 I wrote about Reclaim Hosting experimenting with the next-generation forum software Discourse using a multi-user Docker setup. We use Discourse for Reclaim’s Community forums and I’ve grown to love the software.* What’s more, as that 2015 post notes, it epitomizes some of the challenges of running these next-generation apps on existing, affordable commodity web hosting: namely it runs Ruby using Nginx and requires transactional email. As Tim pointed out yesterday, I never take the easy road with this stuff. Discourse, even on the Reclaim Cloud, is not a simple, one-click install. But I know folks asked me about it, so I wanted to see what the process looks like, and then document it, which I have in the Reclaim Community Forums (which will not be available for another two weeks given we are still building this out, but I will paste in below). I am also working on a video tutorial for this install.

The following guide describes the process of setting up Discourse on Reclaim Cloud, but before I paste it below a note for my particular case that is not necessarily generalizable. When mapping the forum it seems the proxied A record for discourse.bavatuesdays.com through Cloudflare (where I manage DNS for bavatuesdays) was creating issues. I needed to turn that off for Domain mapping to work:

If you are not using Cloudflare this issue should not occur, but it frustrated me for quite a bit.

This guide will take you through installing the next-generation forum software Discourse on the Reclaim Cloud using the official Docker image they maintain.

To begin with you will need to setup a Docker Engine on Reclaim Cloud, by clicking on the downward-facing arrow next to the Docker tab:

After that select Docker Engine:

At the next prompt create the domain (bava-discourse.ca.reclaim.cloud), name the server (Bava Discourse), and decide in what region you want the app to live (blame Canada!).

At this point your Docker Engine server will be spun up, and once it is you can login via the web-based SSH window provided to install Discourse:
From the command line you can install Discourse (and the command line will follow) but before you do you need to ensure 1) you have a transactional email account working and 2) an A record pointed to the container’s public IP address if you plan on using a domain other than mydiscourse.us.reclaim.cloud. In this example we’ll be mapping Discourse to the URL discourse.bavatuesdays.com.

Transactional email services, like Mailgun and SparkMail, allows you to setup email sending and receiving for apps like Discourse. For this example we used Mailgun, and the crucial information you will need are the SMTP server address, SMTP port, SMTP username, and SMTP password. Along with the domain name, this is the information you will be prompted for when setting up Discourse, and if the email does not work (i.e. is not verified through your transactional email service) you will not be able to use the application.

This guide for setting up a new email domain using Mailgun could prove useful. But keep in mind there are more options.

Once that is done you can now begin installing Discourse. From the command line run the following commands:

git clone https://github.com/discourse/discourse_docker.git /var/discourse
cd /var/discourse

After that you can launch the Discourse setup tool:

./discourse-setup

At this point you will be prompted for domain, email, SMTP details, etc.

Hostname for your Discourse? [discourse.example.com]:
Email address for admin account(s)? [me@example.com,you@example.com]:
SMTP server address? [smtp.example.com]:
SMTP port? [587]:
SMTP user name? [user@example.com]:
SMTP password? [pa$word]:
Let's Encrypt account email? (ENTER to skip) [me@example.com]:

We recommend defaulting to port 587 and skipping Let’s Encrypt account email if you do not want to receive email about the built-in SSL certificate.

This will generate an app.yml configuration file on your behalf, and then starts the install which takes a few minutes. If you need to change these settings after installation, you can run ./discourse-setup again (it will re-use your previous values from the file) or edit /containers/app.yml manually with nano and then ./launcher rebuild app, otherwise your changes will not take effect.

Last thing is if you are using a domain other than that provided by Reclaim Cloud you will need to go to Settings–>Custom Domains and add the domain there. Additionally, the domain you are pointing needs to have an A record pointed at the Docker Engine container’s public IP address.

After that, go to the domain and Discourse should be installed and ready to setup.

__________________________

*We are now running Reclaim’s Discourse forum in the Reclaim Cloud quite seamlessly.

Restarting a Discourse Container

We have a server that runs a kind of multisite Discourse environment that I discussed a number of years ago in this post. It is an Ubuntu server with Docker installed, and each of the Discourse instances on that server are spun up in Docker containers. It’s a very small, experimental part of what we do. In fact, we discontinued offering Discourse and Ghost in this kind of environment  a while back, and are far more interested in options like Cloudron, which makes hosting Ghost a breeze. That said, we have a couple of Discourse instances we still host and today the biggest one went down, which is always a bit of a scare for me given it is a unique environment. So, this post is simply going to retrace my steps in terminal to fix this because I always forget given it is not something I do often enough.

When I learned the server was down I figured I would try stopping and restarting the Container to see if that works. To do that I needed to go to var/discourse:

cd /var/discourse

From there, I tried to stop the container (to find the container name I looked in the /var/discourse/containers/ directory which has all the YAML files for each install, and the container names are everything before the .yml extension.

./launcher stop containername

That will stop the container and the following will restart it:

./launcher start containername

But when I went to stop the container I got the a storage full error, and when I ran a

df -h

on the server it was confirmed, the disk was full. I then proceeded to run the trusty NCDU command to get a sense of what was taking up all the space, and I have a suspicion it might be related to this overlay2 storage space issue others have complained about with Docker, but I took the easy route and deleted 10 GBs of old backups for the site and it was immediately back up and running. In the end a restart was not necessary, and I was able to solve a fairly random issue fairly quickly. 

Digital Ocean’s One-Click Apps vs. Cloudron

Digital Ocean has been en fuego as of late. They announced a whole bunch of new droplet plans, and the price-point for all of them has gone down. This is very good news for Reclaim Hosting because it gives us some breathing room with our infrastructure costs allowing us to continue to keep costs low.  We have been slowly moving most of our infrastructure from Linode and ReliableSite to Digital Ocean, and we could not be happier. They are constantly improving their offerings, and being in a virtual environment where we can increase storage or scale CPU instantaneously makes our life (and our clients’) a lot easier.
One-click Apps at Digital Ocean

One-click Apps at Digital Ocean

In addition to new plans and pricing, I noticed they were featuring one-click apps as well (though not sure how new this is), and I took a peak to see what they offered. It was interesting to see that some of the application they featured, namely Discourse (the forum software) and Ghost (the blogging app), were apps Reclaim was offering beyond our shared hosting cPanel-based LAMP stack. Given we’ve been exploring a one-click option with Cloudron (I recently blogged about setting up Ghost using Cloudron) I wanted to compare Digital Ocean’s idea of one-click to Cloudron’s. Long story short, there is no comparison. Here is Digital Ocean’s command line interface for setting up Ghost:

Command line interface during Ghost setup on Digital Ocean’s one-click apps

Here is Cloudron’s:

One-click install of Ghost on Cloudron

Digital Ocean is amazing at what they do, but their idea of one-click installs still assumes a sysadmin level of knowledge, which, to be fair, make sense given they are a service designed for sysadmins. When I tried the Ghost app it was, indeed, installed on a droplet in seconds, but the actual configuration to setup required full-blown tutorial for command line editing the setup. In addition to the domain pointing, this was setting up SSL and Nginx, granted that simply meant typing “yes” or “no” and clicking enter, but even when you did the setup was not guaranteed. After following the tutorial to the letter I still got the Nginx 502 bad gateway error, which means I was stuck.

Ghost 502 Bad Gateway Nginx Error

I could have tried to troubleshoot the 502 error, but at this point it was just a test and from my experience it was far from one-click.

Discourse example

I then tried the Discourse, and this was definitely easier than Ghost. It still required a tutorial, but that was primarily focused on setting up an SMTP account through Mailgun so the application could send email. After that, the setup was simple, but again the one-click setup process on Digital Ocean assumes an understanding of API-driven transactional email services like Mailgun or Sparkpost. Cloudron does not have a Discourse installer, so no real comparison there, but if it could manage the SMTP email setup in the background, I imagine it would be just as simple as their Ghost installer. I’m glad I explored Digital Ocean’s one-click application offerings because it confirms for me the potential power of tools like Cloudron that truly make it simple to install applications. Our community by and large will not be folks with sysadmin level knowledge, so integrating a solution that is truly one-click, avoiding DNS and command line editing,  would be essential.

Investing in Community

Investing in Community

I recently received a support ticket, not so much asking for support, but rather wondering about the status of https://community.reclaimhosting.com/ and why it wasn't promoted more. The person pointed out that it wasn't really linked anywhere or mentioned as far as they could tell, which was absolutely true. When Jim and I first started Reclaim Hosting the idea of building community was very much at the forefront of our minds. I fired up an instance of Vanilla Forums at the time probably only a month after Reclaim Hosting got off the ground. But I never visited there myself. I let it stagnate almost from day one. When Howard Rheingold started experimenting with Discourse I thought "now here's an interesting piece of software for conversing on the web!" and switched up the community site to run on that. I even used a WordPress plugin to make all comments from our blog get driven there as larger discussions. The result: well....nothing. As it not so surprisingly turns out, people don't just flock to new spaces because you hope they will.

The community site has been dormant for a long time now and often I've wondered if it was better to just nuke it into orbit. I had high dreams of folks sharing with each other there, asking questions about how they might approach a given topic, or even user-driven documentation on how to do a particular task on Reclaim. But building community takes so much more than just sitting back and hoping for something to develop. It takes real effort to draw people in, stoke conversations, and it takes a huge amount of good will in the early days. We've had no shortage of good will in building Reclaim Hosting from the community that has embraced us, and if a space to cultivate that is something that I want, something we want, then it's going to take work.

And so last week I rolled up my sleeves and got to work. There were some boring technical details I wanted to accomplish like getting the site to run on SSL thanks to Let's Encrypt support. For the first time I added a link within our client area. I grabbed the RSS feed and started showing the latest posts on our documentation site. And most importantly, I started seeding conversation and inviting folks to the party. You see, Discourse has this great feature that allows you to invite someone to a thread and when they click that link they can immediately start responding without having to go through the process of creating an account. It's a very powerful feature that I have been using a lot this past week to bring folks into the fold and cultivate....well...discourse. Discussions, ideas, tutorials, announcements.

We have a long way to go and it will often require me to get outside of my comfort zone and ask people to participate, be intentional in my actions to seed the space with new ideas and conversation. It's not something I'm used to doing, but it's incredibly important. There is no shortage of amazing people doing incredible work on Reclaim Hosting. And if our support system is any indicator then there are plenty of folks who could use a helping hand as well. We're always there for them, but I would love for that same generosity to extend to the broader circle of people who have trusted us to assist in helping them build their digital identity on the web.

Consider this post me breaking the ice and welcoming you in. I would love for you to come over and chat with us there. As time goes on we'll continue to figure out ways to generate new topics there but I ask that you not be shy and participate in what's happening there. After just one week of investing the time to cultivate the space I've already seen the rewards and it has renewed my efforts to see that space grow. And I now realize this is the investment that we (Reclaim) needs to make in each and every one of you to foster a sense of communal support, the idea that you don't rely on me, or Jim, or anyone else, but that we all can call on each other and have a space to openly share our thoughts. It's more important than ever to me now and I would love for you to join us in that effort!

Investing in Community

Investing in Community

I recently received a support ticket, not so much asking for support, but rather wondering about the status of https://community.reclaimhosting.com/ and why it wasn’t promoted more. The person pointed out that it wasn’t really linked anywhere or mentioned as far as they could tell, which was absolutely true. When Jim and I first started Reclaim Hosting the idea of building community was very much at the forefront of our minds. I fired up an instance of Vanilla Forums at the time probably only a month after Reclaim Hosting got off the ground. But I never visited there myself. I let it stagnate almost from day one. When Howard Rheingold started experimenting with Discourse I thought “now here’s an interesting piece of software for conversing on the web!” and switched up the community site to run on that. I even used a WordPress plugin to make all comments from our blog get driven there as larger discussions. The result: well….nothing. As it not so surprisingly turns out, people don’t just flock to new spaces because you hope they will.

Continue reading “Investing in Community”

Discourse(s) on Docker

One of the things Tim built a while ago is a server running multiple instances of the forum software Discourse using Docker. He did this because we’re getting more and more interest at Reclaim Hosting for this forum software. As usual, Tim came up with a pretty slick setup that enables us to provide this fairly easily and cheaply. To be clear, I have not yet gone through the process of setting up the server environment that runs multiple Docker instances of Discourse, and I want to go through that process next. But in the interim, this post will simply go through setting up a new instance of Discourse using Docker in an attempt to beef up our internal documentation.

If you are interested in getting up and running with Discourse in a Docker container, check out Sam Saffron’s excellent overview of Docker, the various issues installing it, and a write-up of his process, I used that posts on several occasions to be refreshed on the commands I needed to bootstrap and start the docker container.

So, after logging into your server via command line (are you still with me?), you would change directories to where the containers setup files for each server are kept:

cd /var/discourse/containers

 

Discourse 1

You can see in the image above I was searching a bit. In the containers directory you have several .yml files for each of the Discourse installs. YAML files are basically a serialized set of instructions for Docker on how to setup the environment. You have to create a new YAML file for each new install, and ten copy another installs config and change a few details.  So, edit another YAML file by typing something like this:

nano nameofanothercontainer.yml

You will then be shown the config file for that container. Copy it and then create a new YAML file like so:

nano bava.yml

Copy the contents from the other container in this new file, and change the following details.

In this bit you include the email, or emails, of the admins:

## TODO: List of comma delimited emails that will be made admin and developer
## on initial signup example 'user1@example.com,user2@example.com'
DISCOURSE_DEVELOPER_EMAILS: 'info@reclaimhosting.com,jimgroom@gmail.com'

In the following area you change the hostname to the domain (or subdomain) you will be installing Discourse. Chances are you will be mapping an A-Record here. More on that later:

## TODO: The domain name this Discourse instance will respond to
DISCOURSE_HOSTNAME: 'discourse.bavatuesdays.com'

In this area you need to setup your mail server info so Discourse can send emails to new users, etc. We use Mandrill for this at Reclaim Hosting, and it seems to work well. This is a bit difference from applications in a LAMP environment which has all this setup for you. With the new fangled Rudy and Node.js apps you’ll find you need a transactional mail service like Mandrill, Mailgun, etc. —which are basically API driven mail services for developers, or so I’ve heard.

## TODO: The mailserver this Discourse instance will use
DISCOURSE_SMTP_ADDRESS: smtp.mandrillapp.com # (mandatory)
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME:  username
DISCOURSE_SMTP_PASSWORD: yourpasswordhere
#DISCOURSE_SMTP_ENABLE_START_TLS: true # (optional, default true)

Finally, you need to change a few paths to point to your Discourse file. So anywhere you see bava was previously the name of the container’s YAML file I copy and pasted from. For exampled, if I copy and pasted from the reclaim.yml file, everywhere you see bava below would have originally been reclaim

## These containers are stateless, all data is stored in /shared
volumes:
- volume:
    host: /var/discourse/shared/bava
    guest: /shared
- volume:
    host: /var/discourse/shared/bava/log/var-log
    guest: /var/log

Now save the bava.yml file.

After that, we need to edit the discourse settings for Nginx. Notice that Discourse is a Ruby application running on Nginx—two big reasons this application doesn’t run in a LAMP environment. Nginx is a web server, like Apache, but with different requirements than what’s bundled with a LAMP stack. Very few Ruby applications run in a LAMP environment, which means a whole generation of Ruby and Node.js web apps depend on a sysadmin to get running. One of the many reasons to be excited about Docker is that it can potentially make hosting these applications a lot easier. Anyway, we still have to edit Nginx.

Discourse 5

Got to:

cd /etc/nginx/sites-available

From there you need to edit the discourse file:

nano discourse

There will be a series of server settings for each of the discourse containers running. Copy one, and paste it at the end of the file and edit it to work for your container. For example, I replaced the URL I am running my container at (discourse.bavatuesdays.com) as well as putting bava in the proxy_pass file path:

server {

        listen 80;

        # change this

        server_name discourse.bavatuesdays.com;

        client_max_body_size 100M;

        location / {

        proxy_pass http://unix:/var/discourse/shared/bava/nginx.http.sock:;

                proxy_set_header Host $http_host;

                proxy_http_version 1.1;

                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        }

}

Now you need to go to the /var/discourse directory and bootstrap the container and then run it. Below are the commands. When you bootstrap it will take a little while, because that is where an image of your discourse application is being created in the container.

sudo ./launcher bootstrap bava

Discourse 8

If it bootstraps successfully it will tell you as much, and then you can start the application:

sudo ./launcher start bava

If that works, you need to remember to restart Nginx using the command below:

Discourse 11

service nginx restart

We talked about mapping an A-record above, well if you haven’t done that your Discourse application won’t be visible anywhere. So, this might be a good time to go to the DNS Zone editor for the domain you want this to point to and add the IP address of the server to an A-record. It should look something like this:

Discourse 6After that, you should have a brand spanking new discourse application up and running. I still have yet to play with discourse.bavatuesdays.com, so that should be a future post.

Discourse 10

What’s cool is that if I run the docker ps command in terminal, I can see all the containers running discourse.

Screen Shot 2015-08-01 at 10.19.15 PM

If we could automate this process, which I imagine is possible, we could have a server that provides discourse instances for anyone that wants one. Making hosting an application like this relatively easy. And, unlike shared hosting or a multi-site application, the fact that it is in a container means it would not effect any of the others. Trippy and cool.