During a discussion I had with a group of faculty this evening about managing their own domains, one professor asked how he might approach blogging in his film class. Because I can’t help myself, I immediately mentioned to GIFs as a way to capture and analyze specfic moments in a film. A way to pepper a textual analysis with the visual manna. I quickly introduced the Video to Gif tool on Imgur to demonstrate how simple this could be for him and his students. While screen sharing to a room full of faculty I was able to create and embed an animated GIF in less than 3 minutes. THAT is technological progress!
This simple GIF of Danny and Wendy enjoying a wholesome PB&J before all hell breaks loose was created from this Shining trailer on YouTube in seconds. More than anything, this example underscores a domain isn’t necessarily a neat, packaged solution to digital pedagogy, but rather the place you chronicle and archive frequent, free-wheeling jaunts all over the web. And the more GIFs the better.
Two years ago my partner Tim Owens and I rushed down to the Fredericksburg county clerk and gave birth to the two-headed monstrosity that is Reclaim Hosting. I didn’t realize at the time how momentous that would prove 24 short months later. Tim has been full-time for over six months. We hired our first employee, Lauren Brumfield, last month. And I’m going full-time myself, which will allow my family and I to setup shop in Italy for at least a year. All of this made possible by the awesome folks around higher ed and beyond that continue to support the work we are doing. I couldn’t be more excited about where we are right now.
[caption width="500" align="aligncenter"] All Hail the great Michael Branson Smith![/caption]
Last year at this time we had two institutions locked into a Domain of One’s Own project, and two others considering it. This year? We setup up four institutions just this week! And we will have at least 20 schools running a Domain of One’s Own initiative come fall. I’ve been in a bit of a zone the last month or so trying to wrap my head around the Reclaim Hosting infrastructure. I went through the entire process of getting an institution up and running on their own server with only minimal help. Tim has not only carried the load, but he has also architected a pretty amazing experience for schools. Feels good to finally be able to help him out some so he can get back to R&D.
One of the things we’re offering that has me really excited is sane and accessible server/sysadmin support—something that proved invaluable for me when starting at UMW. While I was still getting my head around domains, web hosting, WordPress, etc., I leaned heavily on my friend Zach Davis. I went to that font of knowledge countless times while setting up UMW Blogs. Zach understands servers, web hosting, programming, open source applications, and more. He got me into all this, and he was always very liberal and gracious with his time getting me out of trouble. But, astonishingly enough, ostensibly few ed-tech and digital humanities outfits in higher ed have this kind of support. And none of them have Tim Owens! Reclaim Hosting is all about providing infrastructure and support for those schools that can’t get it locally. And we are doing it at insanely affordable rates. That’s something we pride ourselves on—we want our work to be as accessible to as many people as possible.
The other thing I am excited about is re-thinking hosting in terms of a personalized dashboard for managing one’s digital life bits across time and space. We want to try and re-architect hosting to be both abstracted and deeply personal at once. We’re continuing to chase the vision that has fueled UMW Blogs, ds106, and Domain of One’s Own for years—but we might actually have the people and resources to truly start to build the digital future we want to live in! Happy birthday, Reclaim Hosting. I am a very big Finn fan!
Reclaim Hosting‘s Ramones server was experiencing some extremely high loads this afternoon, and D’Arcy gave Tim and I a heads-up on Twitter to let us know as much. The A-Team was on the job, and more than up to the task! The first thing we check when we’re getting unusually high loads is the Apache Status in WHM (the GUI interface for managing a CPanel server). We look to see if there is one particular site getting hammered with requests—which is often, though not always, the case with random load spikes. In the instance earlier today, one WordPress blog was getting hit very hard with login attempts, often referred to as brute force login attempts. But rathe than the wp-login.php file, it was the xmlrpc.php file which has been a vulnerability for years because it provides, in the words of my server sensei Tim Owens, “a huge target for brute force login attempts because it bypasses the traditional wp-login.php and goes right for logging in via API.” This was precisely the case with the intense load on Ramones this afternoon.
Tim has started collecting snippets of code in our internal documentation, like the one below, that we can just add to the .htaccess file in the affected WordPress install to block all calls to xmlrpc.php. Below is the code snippet we copied into .htaccess this afternoon that brought the load back down almost immediately. Hope you find it helpful. Continue reading “XML-RPC Blocking using htaccess”
Anyway, the image above is a look at a potential setup for a large WordPress Multisite instance on AWS. It has a couple of elements worth discussing in some detail because I want to try and get my head around each of them. The first is a load balancer that runs in its own EC2 instance.
What the load balancer does is direct traffic to the Ec2 instance running the WordPress core files with the least load. So if you have four EC2 instances each running WordPress’s core files, the one with the least usage get the next request. Additionally, if all the instances have too great a load another could, theoretically, be spun up to meet the demand. That’s one of the core ideas behind elastic computing. The load balancer Tim used for UMW Blogs was HAProxy.
As mentioned above, you can setup a series of instances of EC2 on AWS with the core WordPress files save the wp-content directory, which is the only directory folks write to. But you will notice in the fourth instances Tim switched things up. He suggested here that we could have an EC2 instance running Docker that could then run several WordPress instances within it. What’s the difference? And this is where I am still struggling a bit, but from what I understand this allows you to spin up new instances quicker, isolate instances from each other for more security, and upgrade and switch out instances seamlessly. It effectively makes WordPress upgrades in a large environment trivial.
We have yet another EC2 instance that is the Network File Storage, this holds the wp-content files. The uploads, plugins, themes, upgrades, etc. And each of the above instances share this instance. The all write to this, but one of the issues here is that this can be a single point of failure, kinda like the load balancer. So, Tim suggested there is BitTorrent Sync (BTSync), which I still don’t totally understand but sounds awesome. It’s basically technology that synchs files from your computer to spot on the internet, or between spaces in the internet, etc. So, what if we had several bucket where the various instances of WordPress core files were writing the upload files, themes, plugins, etc, and those buckets used BTSync to share between them almost immediately. So then you wouldn’t have a single point of failure, you would have the various instances writing to various buckets of files that would be constantly synching using the technology behind BitTorrent. Far out, right?
Another option, and I think this was before we started talking about BTSync, but not sure if this would be in possible in addition to BTync, is have the blogs.dir folder for a WordPress Multisite that handle all the individual site files uploaded be sent to S3, Amazon’s file storage service.
You get the sense that part of what’s happening when you move an application like WordPress Multisite onto AWS, or some other cloud-based, virtualized server environment, is each element is abstracted out to it basic functions. Core files that are read-only are separate from anything that is written to, whether that be themes, plugins, or uploads. Additionally, the database is also abstracted out, and you can run an EC2 instance on AWS with Docker containers each running MySQL (with a sharDB or HyberDB to further break up load) that can also replicate various writes and calls using BTSync? No single point of failure, and you greatly reduce the load on a WPMS which is completely I’m out of my depth here, but if I I accomplished anything here it might be giving you an insight to my confusion, which is also my excitement about figuring out the possibilities.
I have no idea if this makes sense, and I would really love any feedback from anyone who knows what they are talking about because I’m admittedly writing this to try and understand it. Regardless, it was pretty awesome hearing Tim lay it out because it certainly provides a pretty impressive solution to running large, resource intensive WordPress Multisite instance.
The last two weeks I’ve been spending just about all my time balancing two jobs. My day job at UMW—which is now slowing fading into the horizon—and my amorphous role at Reclaim Hosting as support specialist, burgeoning sysadmin, evangelist, co-founder, and GIFmaker is starting to take center stage. I’ve been working pretty much night and day the last two weeks because my partner, Tim Owens, was on a well-deserved and long-overdue vacation. I won’t lie, I missed him a lot. I have been spoiled. He has single-handedly been running Reclaim like a boss for the last six months. And I got a first-hand glimpse into just how amazingly talented he is while he was at the beach. From support to development to the next idea to bringing more schools onboard, Tim’s the whole package. In fact, it was really intimidating to step in as he was checking out for a while.
Luckily we made the extremely smart move of hiring Lauren Brumfield. What’s cool is over the last two weeks Lauren and I having been working together to get a sense of how everything works together. This will help us tighten some things up, and prepare for the onslaught that will be this fall. We have over twenty institutions working with us this coming semester (ten times the number we had last year at this time), not to mention the countless departments, faculty, and students that Reclaim serves each and every semester. Bringing Lauren on now not only allows her time and space to learn how things work, but also gives us fresh eyes on how we work and what we can improve. We crave constant, focused feedback on how we can streamline some of our service interfaces and operational processes (which we get regularly thanks to many great folks like Christina Hendricks)—so don’t be shy, let us know! Lauren will be working with us to make sure this all happens. We need someone to help us organize, prioritize, and get done many of the things we have outstanding. And given Lauren has taken on the self-appointed title of Operations Manager, and thanks that means we might be getting a bit more orgaminized.
One of the things I worked on over the last two weeks was setting up our payroll. I was in contact with someone from ADP, the payroll behemoth, but Tim and I have been intrigued, and served quite well, by the latest breed of service-based applications coming out to help run a small business. For example, we use the tool Intercom.io for all of our support tickets, and that has been really successful. So after Tim mentioned ZenPayroll, Lauren and I spent some time looking into it over the last two weeks, and the whole design of the tool and the ease and elegance of the on-boarding process for a complete newbie was insanely elegant, helpful, and intuitive. The application not only helps you get your payroll up and running with everything from direct deposits to federal and state tax withholdings to contractor W9s, etc., but it also teaches you how it all works like the best kind of video game. There is a lot to be learned from the design of an app like this.
Tim, Lauren, and I are starting to develop a distributed work routine. A tool that has become central to that routine is Slack. That’s where I talk real-time with Tim and Lauren throughout the week, share issues, ideas, and generally communicate. It allows you to build your own organic channel-based structure, which I love. It doesn’t feel like a system, just a quick, group communication tool with options.
In conjunction with Slack we have another tool call Asana that we use as a project management/to-do list for everything we need to get done. Lauren turned us on to this last week, and we’ve been organizing our various accounts, adding structured to-do lists, and setting up a calendar there. It’s the anti-Slack, and I think the two complement each other well in that regard. So that’s four tools we use to communicate with each other, the innumerable reclaimers, get paid and structure projects.
And none of these deal with a whole other suite of tools that we use to actually manage the various servers, shared web hosting, domains, etc. That list might have to be saved for another post. But what’s been interesting as I witness the distributed architecture of Reclaim emerge, is that it reminds me a lot of the distributed architecture of early Web 2.0 course sites that integrated blogs, wikis, delicious, YouTube, Flickr, Twitter, etc. That radical idea of an ecosystem of course/individual tools is proving just as successful here, but one of the key elements to that success will be making sure the people we help don’t have to struggle between and amongst them—that’s one of the great promises of APIs.
One of the things I think Berman totally nails is the willingness to question the adoption curve of a project like CI Keys. He notes the following:
I am coming to question the usefulness of the innovation diffusion curve in Ed Tech. First of all there’s an implicit value judgment that early adopters are better than late adopters – not to mention the infamous laggards. Not all technology adoption is useful, to say the least, and some is downright harmful. Second, why is success measured as universal adoption? If 20% of the faculty at my campus find CI Keys to be a useful and even transformational tool for encouraging student learning, does that necessarily mean that the other 80% are missing something by not using it? Perhaps, but I’m not so sure. It’s nice to think that we can provide a single tool for everyone to use but we can see where that’s gotten us. Instead, some will use institutional tools, some will use open source, some will use commercial tools, and faculty and students will use different tools (really, media) to accomplish different things. Is that hard from an ed tech support position? No doubt! But I think that’s the world we live in, not one where we always think in terms of scale-up and universal adoption – that ship has sailed.
I can’t possibly have said it better, and it really frames beautifully the predicament at many campuses. Someone throws out the stat that 85% of faculty are using the LMS and the conversation stops there. The various constituents who need resources beyond the LMS are poorly served, if at all. The adoption curve is actually a conformity curve used to justify supporting fewer and fewer tools on campus. So what should be seen as a pretty basic resource like web hosting/publishing are all but absent on many campuses. As Brian Lamb noted in the “Reclaiming Innovation” piece for EDUCAUSE Review:
…institutional leaders may refuse to support alternative systems….lest they draw attention and users away from the “serious” enterprise learning tool, diverting resources and endangering investments. If a technology is sufficiently large and complex, it can dictate policy, resource allocation, and organizational behavior far beyond its immediate application.
And the investment-based logic that can breed an aversion to alternatives often fails to comprehend that they’re not only significantly cheaper than any given system, but often complementary to that system. So, rather than endangering investments, it provides alternatives that make the system that much less monolithic. What’s more, it serves a portion of a campus community that has been forced to fend for themselves for almost a decade.
While folks were bouncing ideas around on how to bring the site up again while still struggling with the outage, I mentioned that I could pretty quickly migrate the site over to Amazon Web Services and run it in Docker containers there. The higher-ups gave me the go-ahead and a credit card (very important, heh) and told me to get it setup. The idea was to have it there so we could fail over to the cloud if we were unable to resolve the outage in a reasonable time.
TL;DR – I did, it was easy, and we failed over all external traffic to the cloud. Details below.
He goes on to describe his process in some detail, and it struck me how the shift in IT infrastructure is moving, and also made me wonder how many IT organizations in higher ed are truly rethinking their architecture along these lines. It’s one thing to push your services to a third party vendor that hosts all your stuff, it’s all together different to bring in a team that understands and is prepared to move a university’s infrastructure into a container-based model that can be hosted in the cloud. Not to mention what this might soon mean for personal options, and a robust menu teaching and learning applications heretofore unimaginable. This would make the LAMP environment options Domain of One’s Own offers look like Chucky from Child’s Play
I know Tim and I are looking forward to thinking about what such a container-based architecture might means for an educational hosting environment that is simple, personalized, and expansive. Tim turned me on to Tutum recently, which starts to get at the idea of a personalized cloud across various providers—something Tim Klapdor gets at brilliantly:
MYOS is very much the model the Jon Udell laid out as “hosted life bits” – a number of interconnected services that provide specific functionality, access and affordances across a variety of contexts. Each fits together in a way that allows data to be controlled, managed, connected, shared, published and syndicated. The idea isn’t new, Jon wrote about life bits in 2007, but I think the technology has finally caught up to the idea and it’s now possible to make this a reality in very practical way.
His post on the topic deserves a close reading, and it’s the best conceptual mapping of what we might build I have read yet. I wanna help realize this vision, and I guess I am writing about Duke University’s move to Docker because it suggests this is the route Higher Ed IT will be moving towards anyway (sooner or later—which could be a long later for some ). Seems we might have an opportunity to inform what it might look like for teaching and learning from the ground floor. It’s not a given it will be better, that will depend upon us imagining what exactly a teaching and learning infrastructure might look like. Tim Klapdor has provided one of the most compelling visions to date, building on Jone Udell’s thinking, but that’s just the beginning.