Archiving ds106 docs

Part of moving ds106 to a new server is making sure you don’t leave a trail of dead links in your wake. With great classes come great responsibility 🙂 I think I have the caching issues and some of the kinks worked out after the move, but one think I did want to make sure wasn’t lost in the move was the ds106 wiki, also known as ds106 docs. It was used through 2014, and while it wasn’t a huge part of the class design, for quite a while we used it to for  tech tutorials, syllabi, and other assorted resources. For example, I forgot about the detailed tutorial I created for an animated series of Dead Zone trading cards:

Or the equally detailed Creating Animated GIFs with MPEG Streamclip and GIMP tutorial.

I understand these resources are not all that useful anymore, but the internet preservationist in me wants them to live on. There are other resources such as various syllabi for classes over the years, such for Alan Levine‘s and Martha Burtis‘s Camp Magic MacGuffin syllabus from Summer 2012, or the syllabus for the ds106 Zone in the Summer of 2013. What I noticed going through my early syllabi for ds106 is they were all the same, they just started riffing on a different theme as the years went by, but the core remained. And while that seems logical, I really didn’t remember simply copying and pasting the basics and then building the theme and the prompts of the class on the blog and through the assignments. So, all this to say keeping the wiki was part of the deal of moving the site off shared hosting.

One thing you realize when moving sites is the value of using subdomains versus subdirectories, let me explain. The MediaWiki instance was installed at ds106.us/wiki rather than wiki.ds106.us. That might have had more to do with the WordPress Multisite being subdomains and my not knowing how to resolve redirects, but if the wiki was installed in a subdomain it would still be live right now (which is probably a bad idea regardless). But given I moved everything in ds106.us and the wildcard subdomains to the Reclaim Cloud, I would not be able to run MediaWiki within a subdirectory of ds106. Whereas subdomain can always be pointed elsewhere, subdirectories lock you into the server you are pointing the root domain to.

So, realizing this I need to a) get the wiki up and running temporarily so that I could then use Site Sucker to get a full HTML-based file backup of the site. This is great for archiving and also ensures that the wiki will not go down as application versions change, modules break, or spammers find a way in.* As you can se from the Site Sucker screenshot above, there are files both in ds106.us/docs and ds106.us/wiki because we used the article path function in MediaWiki to have all articles resolves as ds106.us/docs as opposed to ds106.us/wiki, which explains why the root ds106.us folder has both /docs and /wiki and both have part of the HTML archived files.

Another thing I did before archiving the MediaWiki instance (which I also have a full backup of) was update it from 1.19.xx to 1.33.xx. I had to replace the MonoBook theme, turn off the locked-out module, and adjust some other errors as a result of the update, but I was happy and relieved that it worked after a couple of hours and MediaWiki was now running a supported version on PHP 7.3 no less. Part of me still loves the promise and possibility of MediaWiki, but after wrangling with the documentation and the code it was a good reminder why it was never sustainable-the interface and editing was never made any easier and versioning issues made long-term maintenance onerous.

And with that, I think the future-proofing of the ds106 infrastructure and trying to ensure there remains some link integrity is in pretty good shape. I’ll do another pass this weekend, and then terminate the shared hosting instance, and commit to the cloud!

________________________________________________

*While I was at it I took a flat-file back-up of all of ds106.us and got a database and file dump (as well as a full cPanel backup file) that currently live in DropBox. So, this is a note-to-self that I do have a full snapshot of the site from June 2020 when I go searching for it in the future.

Migrating ds106 to the Reclaim Cloud

If the migration of bavatuesdays was a relatively simple move to Reclaim Cloud, doing the same for ds106 was anything but. Five days after starting the move I finally was successful, but not before a visceral sense of anguish consumed my entire week. Obsession is not healthy, and at least half the pain was my own damn fault. If I would have taken the time to read Mika Epstein’s 2012 meticulous post about moving a pre-3.5 version of WordPress Multisite from blogs.dir to uploads/sites in its entirety, none of this would have ever happened.

I started the migration on Tuesday of last week, and I got everything over pretty cleanly on the first try. At first glance everything was pretty much working so I was thrilled. I was even confident enough to point DNS away from the low-tenant shared hosting server it had been residing on.*

The question might be asked, why move the ds106 sites to Reclaim Cloud at all?  First off, I thought it would be a good test for seeing how the new environment handles a WordPress Cluster that is running multisite with subdomains. What’s more, I was interested in finding out during our Reclaim Cloud beta exactly how many resources are consumed and how often the site needs to scale to meet resource demands. Not only to do a little stress-testing on our one-click WordPress Cluster, but also try and get insight into costs and pricing. All that said, Tim did warn me that I was diving into the deep end of the cloud given the number of moving parts ds106 has, but when have I ever listened to reason?

Like I said, everything seemed smooth at first. All pages and images on ds106.us were loading as expected, I was just having issues getting local images to load on subdomain sites like http://assignments.ds106.us or http://tdc.ds106.us. I figured this would be an easy fix, and started playing with the NGINX configuration given from experience I knew this was most likely a WordPress Multisite re-direct issue. WordPress Multisite was merged into WordPress core in version 3.0, when this happened older WordPress Multi-user instances (like ds106) were working off legacy code, one of the biggest differences is where images were uploaded and how they were masked in the URL. In WPMU images for sub sites were uploaded to wp-content/blogs.dir/siteID/files, and using .htaccess rules were re-written to show the URL as http://ds106.us/files/image1.jpg. After WordPress 3.0 was released, all new WordPress Multisite instances (no longer was it called multi-user) would be uploaded to wp-content/uploads/sites/siteID, and they they no longer mask, effectively including the entire URL, namely http://ds106.us/wp-content/uploads/sites/siteID/image1.jpg.

So, that’s a little history to explain why I assumed it was an issue with the .htaccess rules masking the subdomain URLs. In fact, in the end I was right about that part at least. But given ds106.us was moving from an apache server-based stack to one running NGINX, I made another assumption that the issue was with the NGINX redirects—and that’s where I was wrong and lost a ton of time. On the bright side, I learned more than a little about the nginx.conf file, and let me take a moment to document some of that below for ds106 infrastructure posterity. So, the .htaccess file is what Apache uses to control re-directs, and the those look something like this for a WordPress Multisite instance before 3.4.2:

# BEGIN WordPress
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]

# uploaded files
RewriteRule ^files/(.+) wp-includes/ms-files.php?file=$1 [L]

RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
RewriteRule . index.php [L]
# END WordPress

In WordPress 3.5 the ms-files.php function was deprecated, and this was my entire problem, or so I believe. Here is a copy of the .htaccess file for WordPress Multisite after version 3.5:

RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]

# add a trailing slash to /wp-admin
RewriteRule ^wp-admin$ wp-admin/ [R=301,L]

RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
RewriteRule ^(wp-(content|admin|includes).*) $1 [L]
RewriteRule ^(.*\.php)$ $1 [L]
RewriteRule . index.php [L]

No reference to ms-files.php at all. But (here is where I got confused cause I do not have the same comfort level with nginx.conf as I do .htaccess) in the nginx.conf file on the Reclaim Cloud server there is a separate subdom.conf file that deals with these re-directs like so:

    #WPMU Files
        location ~ ^/files/(.*)$ {
                try_files /wp-content/blogs.dir/$blogid/$uri /wp-includes/ms-files.php?file=$1 ;
                access_log off; log_not_found off;      expires max;
        }

    #WPMU x-sendfile to avoid php readfile()
    location ^~ /blogs.dir {
        internal;
        alias /var/www/example.com/htdocs/wp-content/blogs.dir;
        access_log off;     log_not_found off;      expires max;
    }

    #add some rules for static content expiry-headers here
}

[See more on nginx.conf files for WordPress here).]

Notice the reference to WPMU in the comments, not WPMS. But I checked the ds106.us instance on the apache server it was being migrated from and this line existed:

RewriteRule ^files/(.+) wp-includes/ms-files.php?file=$1 [L]

So ds106 was still trying to use ms-files.php even though it was deprecated long ago. While this is very much a legacy issue that comes with having a relatively complex site online for over 10 years, I’m still stumped as to why the domain masking and redirects for images on the subdomain sites worked cleanly on the Apache server but broke on the NGINX server (any insight there would be greatly appreciated). Regardless, they did and everything I tried to do to fix it (and I tried pretty much everything) was to no avail.

I hit this post on Stack Exchange that was exactly my problem fairly early on in my searches, but avoided doing it right away given I figured moving all uploads for subdomain  sites out of blog.dir into uploads/sites would be a last resort. But alas 3 days and 4 separate migrations of ds106 later—I finally capitulated and realized that Mika Epstein’s brilliant guide was the only solution I could find to get this site moved and working. On the bright side, this change should help future-proof ds106.us for the next 10 years 🙂

I really don’t have much to add to Mika’s post, but I will make note of some of the specific settings and commands I used along the way as a reminder when in another 10 years I forget I even did this.

I’ll use Martha Burtis‘s May 2011 ds106 course (SiteID 3) as an example of a subdomain migrated to capture the commands.

The following command moves the files for site with ID 3 (may11.ds106.us) into its new location at uploads/sites/3

mv ~/wp-content/blogs.dir/3 ~/wp-content/uploads/sites/

This command takes all the year and month-based files in 3/files/* and moves them up one level, effectively getting rid of the files directory level:

mv ~/wp-content/uploads/sites/3/files/* ~/wp-content/uploads/sites/3

At this point we use the WP-CLI tool do a find and replace of the database for all URLs referring to may11.ds106.us/files and replace them with may11.ds106.us/wp-content/uploads/sites/3:

wp --network --allow-root search-replace 'may11.ds106.us/files' 'may11.ds106.us/wp-content/uploads/sites/3'

The you do this 8 or 9 more times for each subdomain, this would obviously be very , very painful and need to be scripted for a much bigger site with many 10s, 100s or 1000s of sub sites.†

To move over all the files and the database I had to run two commands. The first was to sync files with the new server:

rsync -avz root@ds106.oldserver.com:/home/ds106/public_html/ /data/ROOT/

Rsync is is the best command ever and moves GBs and GBS of data in minutes.

The second command was importing the database, which is 1.5 GBs! I exported the database locally, then zipped it up and uploaded it to the database cluster container and then unzipped it and ran the database import tool, which takes a bit of time:

mysql -u user_name -p database_name < SQL_file_to_import

After that, I had to turn off ms_files_rewriting, the culprit behind all my issues. That command was provided in Mika’s post linked to above:

INSERT INTO `my_database`.`wp_sitemeta` (`meta_id`, `site_id`, `meta_key`, `meta_value`) VALUES (NULL, '1', 'ms_files_rewriting', '0');

You also need to add the following line to wp-config.php:

define( 'UPLOADBLOGSDIR', 'wp-content/uploads/sites' );

The only other thing I did for safe-keeping was create a quick plugin function based on Mika’s stupid_ms_files_rewriting to force the re-writing for any stragglers to the new URL:

function stupid_ms_files_rewriting() {
$url = '/wp-content/uploads/sites/' . get_current_blog_id();
define( 'BLOGUPLOADDIR', $url );
}
add_action('init','stupid_ms_files_rewriting');

I put that in mu-plugins, and the migrated ds106.us multisite install worked! There was some elation and relief this past Saturday when it finally worked. I was struggle-bussing all week as a result of this failed migration, but I am happy to say the Reclaim Cloud environment was not the issue, rather legacy WordPress file re-writes seemed to be the root cause of my problems.

I did have to also update some hardcoded image URLs in the assignment bank theme , but that was easy. The only thing left to do now is fix the ds106 MediaWIki instance and write that to HTML so I can preserve some of the early syllabi and other assorted resources. It was a bit of a beast, but I am very happy to report that ds106 is now on the Reclaim Cloud and receiving all the resources it deserves on-demand 🙂


*VIP1 was the most recent in a series of temporary homes given how resource intensive the site can be given the syndication hub it has become.

†I did all these changes on the Apache live site before moving them over (take a database back-up if you are living on the edge like me), and then used the following tool to link all the

ds106radio Chats

One of my favorite things to come out of this time of social distancing has been the revival of ds106radio. Its been such a treat to jump online to listen to music and conversations!

I was first on the radio back in 2014 when I took wire106. Classmates and I create a group project to produce/record/edit our own radioshow. These episodes were all centered around the Wire. I’ve got an archive of the class (sans images which is a whole other story) but it’s worth a look through!

So as Jim’s been streaming he started brining on each of us at Recalim to chat about things and projects they’ve been working on. He’s had the ever amazing Lauren Brumfield on a few weeks ago and they had an awesome conversation.

Then it was my turn on April 24! Jim and I chatted about hot topics in the world today and my past in ds106, as well as contemplating culture and support within Reclaim.

I’m super grateful Jim recorded this convo so I can look back on it to see where I’ve been and where I’m going at Reclaim. It’s a good marking point within my career and a great way to take the pulse of how my year has gone.

I’ve uploaded the hour convo up to SoundCloud but stayed tuned for an even longer one that Jim posts!

Header photo by Anmol Arora on Unsplash

Back in the Burg

I’ve been back in the Burg for the last 10 days working from Reclaim’s HQ on a range of stuff. As a result the bava has been a bit quiet given the push to get as much done as possible in a relatively short time frame. I’ve been able to get some of my stuff out of storage on the first day of my return, which has been on focus of the trip. I’ve been going through boxes of toys, books, movies, and more which is always a fun past time for me. I’m figuring out how to get my stuff overseas in the next month os so, but until then I am using CoWork’s unclaimed spaces as a temporary waylay station. 

This rip-up brought to you by Civic TV

Paint it Black

We have also been working quite diligently on making Reclaim Video a reality, which has been quite a blast. I’ll post more on that soon, but we did a pretty intense carpet and tile rip as well as began painting the store, which Lauren blogged about earlier today. Watching the space come together has been a dream come true—I’ve pretty much wanted to run a VHS store since I was a pre-teen, so this is pretty exciting.

Pioneer DVL-700

I have also been doing some shopping for VHS tapes, laserdiscs, game consoles, and more. I went to Fat Kat Records 20 minutes south of here in Ladysmith and picked up a ton of mint laserdiscs as well as a mint Pioneer DVL700. I even tested it out with a showing of Red Dawn …. WOLVERINES!

Red Dawn Title

When the Mongols could see each other they had worked themselves up into a pretty good frenzy.

I’ve also been a regular at the Library of Congress’s Packard Campus in Culpepper, VA, which has been amazing. I got to see Sense and Sensibility, Close Encounters of the Third Kind, The Last Picture Show, and Lawrence of Arabia , all of them in glorious 35mm. I even missed a few gems like Guess Who’s coming to Dinner, A River Runs Through It, and The Adventures of Baron Munchausen—world enough and time!

Packard Campus February Program

But that’s alright, there’s always March!

Packard Campus March Schedule

I was a little sad to learn that Culpepper’s 1938 State Theater had closed after only being re-opened for two years. There was a major funding drive to get investors help refurbish and re-open Culpepper’s movie house at the tune of $13 million dollars, It was an impressive remodeling to restore it to its original glory. I had the good fortune to see Independence Day there in 2014. But that was then, since it has gone defunct and just a few weeks ago it was auctioned to the highest bidder for $700,000.

I was also able to rekindle my ds106 roots with a quick stop by UMW yesterday to record a  video for The End 106 with the great Martha Burtis. She’s a genius.

 And later that afternoon I actually got back in the classroom after a long hiatus to talk to to an awesome group of students in Eddie Maloney’s graduate course Technology Innovation by Design, which is part of Georgetown’s new Masters program in Learning and Design. It was a thrill to talk to student who want to think critically about the future of educational design, and I’ll write more about my approach in a follow-up post. I do miss the classroom, it is always a lot of fun for me—but damn I tend to talk a lot.

Anyway, if nothing else, this post serves as a roadmap for all the posts I need to write after taking a bit of a hiatus from the blog in order to dig in a bit while here in the US. it’s been quite nice to work alongside Meredith, Lauren, and Tim in CoWork—it’s been a welcome change to reconnect in person with the awesome crew that makes Reclaim so damn good.

Reclaim’s Fantastic Four

Yesterday was a special day, Reclaim Hosting grew to four full-time employees with the hiring of Meredith Fierro. Four is the perfect number for a super group, wouldn’t you agree? Needless to say, the costumes are on order. Meredith worked as an intern in Spring, a part-time employee over the Summer, and as of yesterday we made it official. Meredith will be focusing on support, and she is already bringing some of that Digital Knowledge Center magic to Reclaim. Our timing was perfect given this Fall has been quite busy, but I believe we’re handling it like, well, spandex-clad super heroes.

Continue reading “Reclaim’s Fantastic Four”

Reclaim the Office

Lindley Estes caught up with Tim Owens yesterday to talk about the new Reclaim Hosting offices in Fredericksburg. Tim and Lauren have already written about the space, and just today Lauren posted an update with pictures of the demo work going on as well as some insights to our thinking about the redesign.

On top of that, Lindley published an article for the Freelance Star today based on her conversation with Tim. What was cool about the article is how she connected the new offices with our earlier collaborations at UMW. Tim’s first day on the job at UMW in 2011 was being thrown into a nutty office space with a makeshift TV studio and a missing Dr. Oblivion, as well as hysterical teaching assistant-once-removed Martha Burtis. The fabled Summer of Oblivion!

When Tim and I were still at UMW one of the things we talked about was having a space like the DTLT offices for Reclaim. Three years later that is becoming a reality, and that’s truly awesome. As noted in the article and Lauren’s post, we’re opening the space up, exposing the ceilings. adding a glass-enclosed conference room, laptop bar, collaborative table, private desks, booths, as well as a recording studio, green screen, and 3D printing. We even have awesome sonic professionals folks like Mark Snyder who have already offered to help out.

That said, I’ve seen enough to know posh offices don’t necessarily make the magic, that’s about the people, the vibe, and the freedom. And after our trip to Portland, it was clear we have those three right now—in fact it was being together for that week that made us think about doing this. We are all locked-in, and very much taking care of business as I suggested in a recent braggy post about just this topic. In fact, Brian Lamb commented on that post asking how we managed our distributed workflow, which brought me to the post I wrote over a year ago about settling into distributed work from Italy. All of which seems ironic given I am writing a post about our new collaborative office space in Fredericksburg, right?

Maybe, but it all seems related to me. We know we can effectively run Reclaim Hosting in a distributed manner around the globe—and I’ll remain distributed for the foreseeable future. But the idea of creating a dynamic space that can become a headquarters for Reclaim as well as a communal hub for distributed workers in and around Fredericksburg is a new challenge. And if it has any of the energy and goodness of ds106, as Lindley suggested in her Tweet above, then I can’t help but think it will be a most fun and creative one at that.

Muhlenberg College: The ‘Berg Builds

This post has sat in draft for almost three weeks while I finished traveling and decompressing from a month on the road. Like Cogdog, I have been a tired blogger these days, but unlike Alan I haven’t always been able to push through it. But maybe that’s not all bad, because during that down time there have been a number of other posts about the event at Muhlenberg by the people who are making it happen! Blogging is everything, indeed!  I would encourage you to check those posts out, and thankfully Lora Taub-Pervizpour provided an awesome round-up post with links to many of them.

Continue reading “Muhlenberg College: The ‘Berg Builds”

Some Hosting Advice for Reclaim

Hosting Advice feature on Reclaim Hosting

A month or so ago Alexandra Leslie of Hosting Advice reached out to us to find out a bit about who we are (i.e. Reclaim Hosting) and how we got started. Turns out they got wind of this small, niche hosting company that predominantly serves educators and students, and they were intrigued. While I was traveling in the UK they published a feature blog post about Reclaim Hosting that recounts our story—and I have to say it was pretty cool to read it. I think we’ve been so deep in the day-to-day of running Reclaim that it’s easy to forget there is a pretty cool story arc developing around this work, and it includes a broad community of folks that want to take ed-tech back from the venture capitalists, data fascists, and boring ass learning management systems.  Continue reading "Some Hosting Advice for Reclaim"

CPanel’s Transfer Tool

Recently I was moving a shared hosting client from one of our older servers to a newer one because of some performance issues she was having given the resource needs of her site. I used CPanel’s Transfer Tool for this, and I wanted to quickly note how powerful it is. The Transfer Tool enables you to move an entire cPanel account from one server to another with a few clicks. When migrating accounts from other hosting companies that use CPanel, we often use cPanel’s Restore a Full Backup tool (another clutch feature) because it allows us to move all the files, test to make sure everything works, and then point the DNS. But when moving an account from one Reclaim Hosting server to another, we usually use the Transfer Tool. It’s simpler because both sites are within the same DNS Zone and we know our servers won’t block the request, which is not the case when pulling accounts from other hosting services. Finally, it is dead simple.

As an example, I decided to move the venerable ds106 from the Minutemen server to the Unwound server for the sake of getting screenshots and narrating this post.
Continue reading “CPanel’s Transfer Tool”

Coffee and #ds106 at FredXchange

CYw1qiyWcAAUQyn I am back in Fredericksburg for a few days for some business and house cleaning. I head up to NYC on Tuesday, so it will be a fairly quick turn here, but it's always exhilarating to spend some quality time with my pardner Tim Owens. Yesterday was particularly eventful because we we're invited to talk at an Open Coffee Event hosted at the Foundry, the new co-working space  in Fredericksburg brought to you by the good people of the FredXchange. 12407529_203904386621759_989721171_n The story of how this came about is another bizarre testament to the wonder of the web. Libby O'Malley, one of the movers and shakers behind FredXchange, emailed me out of the blue a few weeks back asking me if I wanted to talk with the folks at the FredXchange about the venerable and righteously right ds106. She informed me that she had found out about that international gem through doing a google search for "Digital Marketing Degree," and ds106 was mentioned as a free alternative in an article that was a top-hit . I think it may have been this article, but I'm not sure. Look ma, no SEO!!! Crazy, I told her I am in Italy living like a pimp daddy in the old country, to which she suggested I come in via Google Chat. As much as I love the internet, I hate remote presentations. They can be done well, but the amount of work it takes to create the conditions of a good remote presentation versus simply being there is staggering. That said, I did know I would be in Freddy for a few days and threw that out and she agreed. I'm glad she did. 12547195_217760811892155_1759698265_n I have been in repose in my mountain villa in Italy mustering as much thought leader mojo as possible. It's hard work, and I don't recommend it for the faint of heart. Being a visionary is extremely demanding, especially in a country that is steeped in the hairy legacy of Lucio Battisti. But ed-tech pioneers persevere, and I knew it was time to get back on the road after 3 months of intensive meditation. And what did I do? I got nostalgic about the past---but I blame that on my new surroundings. More seriously, I hadn't presented for a few months, which is a good while for me. It felt good to talk about ds106. It just never gets old, and once I get going I feel as passionate about it as I did in 2011. It was pure in my heart. Tim and I didn't have a presentation prepared, or even a plan really. We talked briefly about an outline on the car ride over. Early days of the Bluehost experiment at UMW -> UMW Blogs -> ds106 -> Domain of One's Own -> Reclaim Hosting. It's interesting to look at that narrative progression and to see ds106 at the center of the story. I think that's pretty accurate. ds106 galvanized a community, brought Tim and I into contact, and reaffirmed that the Domain of One's Own experiment wasn't all that crazy at scale. I took the first 10-15 minutes talking about the Blue Host Experiment, UMW Blogs, and the beginnings of ds106, and Tim took the last 10-15 minutes talking ds106, Domain of One's Own, and Reclaim Hosting. 10413901_1645532262375051_656298805_n-1 It was quite fun, and I really enjoyed being back in front of a group of people talking about this stuff with Tim. One of the things about my work with Tim that is so rewarding is that we truly do have an amazing partnership. We agree on the fundamental principles undergirding Reclaim Hosting: fierce independence and freedom online! What's more, we push each other to keep going further. Tim has an insane standard for the quality of work and support that defines Reclaim, and every day I wake up knowing that it is the goal. When people come to Reclaim they get the best ed-tech support anywhere. Period. I stand by that statement. Try me, I dare you. It was awesome to be back in Fredericksburg hanging with Tim Owens sharing the work we've done thus far and planning for what's to come. In short, taking care of dot.bizness.