The Evolution of the Cloud

We have some really exciting things that we're working on right now that we'll be sharing in the coming days and weeks ahead. But I think it's worth first taking a look back at my personal and Reclaim Hosting's  history in hosting to understand why I think this next step is such an important one. This will be a long post, but anyone who has been watching or has been a part of this history will likely appreciate the context for how these ideas have evolved (I know I personally do, and after all that's what this blog is for).

Shortly after I joined DTLT at the University of Mary Washington in the summer of 2011 one of my first projects was to start DTLT Today, a semi-regular video podcast that everyone in the group to some capacity participated in. We streamed it live which provided a really cool synchronous element and I still go back and watch the shows today to get a sense of where our heads were at around particular topics at the time. You can find all the episodes on my YouTube channel if you scroll back far enough. And that streaming part was an interesting one. I had experience through ds106.tv with third party tools like Justin.tv (now defunct because they became Twitch) and it was before the days of YouTube offering free streaming. I got it in my head to play around with Amazon Web Services to quickly create a streaming server of our own running streaming software called Wowza. The software would have been quite expensive and I surely had no server of our own on campus I could run it on, but for just an hour of live streaming we could do it very cheap (pennies per day).

Streaming Live Video without Ads for Pennies
I’ve been in search of a decent low cost video streaming solution for a long time now. It doesn’t take long playing with Ustream [http://ustream.tv] and Livestream [http://livestream.com…
The Evolution of the Cloud

That was my first time experiencing what "the cloud" could really do for web-based projects. The distributed plug and play nature of the tools got me hooked because there was a very low barrier to entry. If something wasn't working right, destroy it and start a new one. There was a marketplace of applications I could rely on for prebuilt stuff and even back in 2011 AWS had a lot of different services (now exponentially so).

By 2014 we had started to turn our eyes to AWS once again as a possible alternative hosting environment for UMW Blogs. We had suffered performance issues on that platform for quite awhile as we tried to handle growth and this seemed an opportunity for us to have a more flexible infrastructure that could grow with that project. We had planned to just move the database to an RDS instance and maybe the uploads to S3. But we ran into such latency that rather than scrap everything we added an EC2 server in and moved the whole application. And it was so fast!

UMW Blogs in the Cloud
Anyone tasked with running a large WordPress Multisite install at an institutional level has likely dealt with their fair share of issues in the past few years from database scaling with large growth to the constant barrage of spam and login attempts that can DDOS and server and bring it to its knee…
The Evolution of the Cloud

In the Fall of 2014 I had also begun playing with a piece of software I was starting to hear about called Docker. Something about servers being the old guard and "containers" being the new hotness. But what was a container? On the face of it they felt a lot like virtual servers. There was a hub I could search and find all kinds of different applications just like at AWS. But these were much faster than spinning up an EC2 instance because containers didn't have to start an entire operating system and make use of all the hardware of the computer they were running on, they were much smaller virtualized instances of just the application they were running with the ability to share all the common OS-level stuff across multiple containers.

Keep in mind Reclaim Hosting had been a thing for a year at this point and Domain of One's Own at UMW for 2 years. Add to that Hippie Hosting and I had about 3 years of server experience under my belt but it was all in the context of shared hosting with Plesk and cPanel. Because Docker allowed me to run applications I could never run previously in LAMP servers I was immediately hooked. I figured out a way to share multiple Ghost containers on a single server so we started offering that through Reclaim Hosting. Similar with Discourse forum hosting. I even had a Federated Wiki instance up and running for a time that was running on Docker containers (remember that?!). It's been a long running dream for us to offer an elegant solution to allow people to play with this stuff. But there was still a lot of manual work creating and removing these containers when users would sign up, and a lack of a real solution meant slow adoption and difficulty supporting it.

Fast forward to the end of 2014 and I've given notice that I'm going to do Reclaim Hosting full time, quite a leap of faith given the security of working at a public institution. But this shit was fun, people were trusting us, and the community was growing. In December of that year Jim and I had the thought to bring Kin Lane to Fredericksburg to talk to us about APIs and think through how we might build an architecture for that at Reclaim Hosting. My head was still in the clouds with all the Docker stuff and we couldn't resist thinking about it in terms of containers, all different endpoints and applications as various containers with their own unique hostnames. It was a solid few days of brainstorming that pushed the very boundaries of my understanding at the time and I still treasure not just these scribbled notes on a whiteboard but also the generosity of Kin's time with us.

The Evolution of the Cloud

In early 2015 I learned of a project called Sandstorm that allowed for container-based hosting of applications. It was open source and I set a server up to start playing. It checked a lot of boxes in making it easy to start containers running Ghost, Etherpad, and other apps without the overhead of needing to be a developer, but the drawbacks were that it was not using Docker and building apps to run in the environment was very difficult with their model. It also seemed pretty heavily to favor collaboration applications with less of a focus on publishing so oddly content management systems were more difficult to work with in their model. The group developing Sandstorm got hired by Cloudflare and the project is mostly abandoned at this point, but I know OpenETC has made quite a bit of use of the platform.

The Coming Sandstorm.io

By August of 2015 I return back to AWS again, this time to experiment with moving Reclaim Hosting's main site to a distributed stack there. Not just a single EC2 instance with files in S3, but rather a load balanced stack with multiple app servers in different regions, a CDN on top, and deployment through Git and a staging server to boot. This was amazing but it also ended up being quite expensive and a lot of overhead to manage. I didn't find Amazon's tools very user-friendly (there are much more developer-focused of course) and I couldn't justify the high costs of multiple EC2 instances for a site that while mission critical also wasn't a massive load at the time even on a shared server.

Reclaim Hosting in the Cloud
This past week marked the 2 year anniversary of Reclaim Hosting [https://reclaimhosting.com] and what started as something of an experiment has turned into a successful business and one of the most rewarding things I’ve been a part of in my professional career. We’ve come a long way in 2 years but w…
The Evolution of the Cloud

3 years ago in Fall of 2017 I learned of a project called Cloudron and started playing with it. Similar to Sandstorm you could install it on your own server, it was (key word was) open source and this system used Docker containers at its core. Even more it made it fairly easy to map custom domains, provision SSL certs, and run your software on the web with ease and the list of support apps had a lot of amazing stuff like Gitlab, Ghost, Etherpad, and more. We piloted a program with Bates College in which we integrated a Cloudron server with their Domain of One's Own program and we even considered at one point whether or not Cloudron could be the backend of some kind of future "Domains 2.0" environment where folks weren't limited by the LAMP stack. Unfortunately what we found was that the focus on Cloudron as a personal server made it less of a great fit for a multi-tenant model. There was a basic user management built in, but most models reflected a single or multiple admins handling all the installation rather than a true platform as a service style offering that we were hoping for. And the business model of Cloudron has changed a lot over the years with open source now being nothing more than a label (turns out the ability to install applications is not open source and requires subscription fees paid to them so there's a lot of lock-in there) and development of our own applications was proving to be challenging because they supported Docker but you had to build with the container as a base and setup volumes in a very particular way for them to work. So back to the drawing board.

Beyond LAMP
Since Reclaim Hosting was founded in 2013, cPanel and the traditional “LAMP” stack have been at its core. LAMP is an acronym for Linux, Apache, MySQL, PHP and some of the more familiar applications you love like WordPress and Omeka run that tech stack. This comes with its own limitations as newer so…
The Evolution of the Cloud

The common themes here are that this stuff is not new (nothing ever is) but that it has traditionally been very complex and framed for developer audiences. And in lot of ways that's where Reclaim Hosting found themselves with the state of web hosting in 2013. It wasn't that cPanel or LAMP stacks were some brave new world we were exploring, but we saw an opportunity to build a community around providing easy hosting for educators in a context where you could get real support and build out a presence on the web. We found our niche in that and have continued to double down on that with our Domain of One's Own program and Managed Hosting services. The platform plays a big role, but so do the people.

Domain of One’s Own: Notes from the Trailing Edge

Maybe you're building your Next Generation Digital Learning Environment and wanting to use open source tools like Mattermost, Etherpad, Jitsi Meet, and WordPress to create a framework for your courses. Maybe you're like me in 2011 and wanted to play with some complex piece of software without investing in the high server costs and overhead of buying a VPS (ever tried to run your own install of Canvas?). Maybe you're curious to try running your blog on Ghost like this one here. And maybe you want to start a project small with potential for it to grow in a big way over time and you don't want to have to continually adjust hosting environments for whatever level will account for new traffic you are experiencing (we see that a lot with WP Multisites on campuses where year 1 looks very different from year 5 of the project, or hell even a Sunday evening when everyone's homework is due). Is there a DH project that you've been eyeing but not sure how to install and run it? Are you trying to get support for newer scholarly publishing platforms at your university but you aren't sure where to start?

Our efforts and experiments throughout all of this have been to answer the question of "what's next?" and look towards the future of web hosting with an evolution of the past. It needs to have a user-friendly interface. It needs to be affordable. It needs to be without limitations. It needs to scale.

Today starts that journey, and Reclaim Cloud is coming very soon. In the coming weeks we will be talking more and more about what this platform is and what it can provide. If any of this post hits on struggles you've had or interests please do add your email to the list to be updated and we'll be working to add more people to the beta as we ramp up towards a public launch.

Reclaim Hosting in the Cloud

Reclaim Hosting in the Cloud

This past week marked the 2 year anniversary of Reclaim Hosting and what started as something of an experiment has turned into a successful business and one of the most rewarding things I've been a part of in my professional career. We've come a long way in 2 years but we're learning everyday and one thing on my ever-growing bucket list of bugs and improvements has been our website. Not the design mind you (though I do have lots of plans to give that a makeover), rather the architecture and hosting of it.

When we started the company we had just one lonely server for people's sites and that included our own. Today I manage many servers but in part out of laziness I never did move our site off the same shared hosting server our customers were on. I could justify this for awhile because it's not like we get a ton of traffic (though we're not small), but also I felt at the time it was a good gauge for our service if we were on the same system everyone else was using. Alas, I started to see holes in that methodology when people who had various issues with accessing their account be it firewall issues or otherwise would also no longer be able to get to our site. More and more it seemed like reclaimhosting.com should always be up and always be available regardless of the status of any given server we manage.

Meanwhile for the past year I've dug more and more into Amazon Web Services for server architecture. I helped UMW move UMW Blogs to AWS last summer and in fact the DNS for reclaimhosting.com has been running through Route53 for at least a year now. With a few recent projects being candidates for a multi-layered cloud approach with AWS I thought it was high time we moved the main website for Reclaim Hosting up there as a proof-of-concept and best-practice example of running WordPress in the cloud.

I started with the video below (the audio for it is incredibly quiet but it's worth it) which helped me figure out a few neat tricks for caching, staging environments, and other things I hadn't yet experimented with. The author also shares a lot of information in the notes that point to his GitHub repo for the project.

Today I flipped the DNS and at this point I believe the site is now resolving for most folks on the new setup. Here's a quick laundry list of some of the items at play with our new setup for reclaimhosting.com:

  • A single EC2 server serves as a staging environment that utilizes the same database as production servers.
  • The staging environment changes are committed to a private repo on GitHub (mostly just plugin and theme modifications since the database stores most all content)
  • Our database is using a Multi-AZ (Availability Zone) RDS instance for high availability.
  • We have an Elastic Load Balancer that is receiving the requests to the site and sending them to production servers.
  • OpsWorks is setup to deploy changes from the git repo to 4 production EC2 servers.
  • Each production server is located in a separate datacenter for high availability.
  • All uploads are sent to S3 storage and served on the website using the CloudFront CDN for faster response times globally.
  • A Redis object cache stores objects across all production servers in memory on an ElastiCache instance.

Reclaim Hosting in the Cloud

The new setup also has me digging a bit into Chef which is used in the deployment process by OpsWorks but not something I previously had any experience with. The upgraded setup also comes with a lot of security enhancements. Logins are only available via white-listed IP to our staging environment and wp-login.php is completely removed during deployment so it's virtually impossible for someone to get in to our site. Security groups (essentially firewalls) are configured in a way that each layer only has the necessary ports open to the necessary servers accessing them. For example the database can only be accessed by the running EC2 servers, the production servers will only access ports 80 and 443 (http/https) from the load balancer, and staging is only accessible via SSH with a private key.

I started out with T2-Micro instances across everything (the cheapest Amazon offers), bringing the total cost of a setup like this to around $60/month running 24/7 with 5 servers, a production database, an in-memory cache instance, and ~1GB of data stored in S3 with CDN caching. At the moment performance seems to be pretty great considering the small size of the production servers.

This is no doubt an incredibly complex environment and not something I would recommend anyone tackle for their personal sites, but for large production instances of WordPress used by your campus or sites that absolutely need to be online at all times and highly-available globally? Absolutely! And heck, if you're unsure how you might accomplish the same hire Reclaim Hosting to help you get setup or host it all for you in the cloud!