Forum migration

this post documents the migration process of this forum from discourse hosting to private hosting, for the following purposes:

  • keep a track record for my own sanity
  • update the forum members of the progress, and give some background info for the curious
  • receive feedback and suggestions from those members who know a thing about hosting (I know merely enough to be dangerous…)
  • leave breadcrumbs for those will walk after us (e.g. anyone in the future wanting to migrate a hosted discourse forum to a private server)

Lets start with the basics: This forum uses the Discourse software, developed by the fie people at discourse.org. I have chosen this software to host our community forum primarily for its smooth user interface (be the judge, you are using it right now) and its intelligent trust-v-based permission management.
The software itself is available for free. Discourse also offers a hosting service, which is awesome, but not cheap. After running this forum hosted on discourse for many years, I have decided to reduce my costs by skilling up and hosting this forum myself.

The journey begins here:

5 Likes

Discourse suggests DigitalOcean as hosting provider.
DigitalOcean offers Droplets in its Marketplace, which I understand as preconfigured packages of OS and apps/frameworks.
The Discourse droplet contains Ubuntu as OS and has Discourse and Docker CE installed.

2 Likes

I dont run my own mail server. But I need one…
I set myself a time limit of 30 minutes and do some research on mail providers. (If I dont inflict a time limit on myself on research tasks, I tend to down the rabbit hole to deep…)

I decide to go with Mailgun. A lot of positive reviews, and it also seems to be a popular choice for Zapier, which I love using for all sort of tasks, so it would be nice to extend Zapiers email capacities with Mailgun.

First lecture on mailgun: the domain verification process:

My main website including the domain is hosted at Shopify.
So I go into the admin interface, sales channels -> online store -> domains
and add two TXT records as per Mailgun documentation:

In the Mailgun admin interface, I click “verify DNS setttings”, but I see red crosses next to the TXT records:

Im not sure if Im doing it wrong, or if it needs some time to propagate.
I will interrupt my work at this point and get a few other things done.

1 Like

While it does takes some time to propagate DNS changes (in my experience TXT are especially low update frequency). it looks like you put the SPF key on the root of the domain:

;; ANSWER SECTION:
micro-motor-warehouse.com. 21599 IN     TXT     "v=spf1 include:mailgun.org ~all"

which also matches your screenshot, the “@” denotes the root (== no subdomain), this shold be “mail”.
the actual key record i could not dig up at all. make sure the name part in the interface says “mx._domainkey.mail

Also make damn sure you got every character of that RSA key. i ran in that one myself…

3 Likes

Awesome, one step ahead!
I had the feeling Im doing it wrong… you led me on the right track.
I changed the first TXT record name from @ to mail, and mailgun picked it up straight away:


I shortened the second TXT record (where I previously had the complete domain in there) to mx._domainkey.mail, and a minute later Mailgun was happy and shows me this screen instead of all the instructions:

Sweet, moving on :smiley:

1 Like

Next step: creating the DigitalOcean droplet. After clicking “create”, there is a little shark swimming along.


I like it :smiley:

It asks for the details of the machine:

I guess the minimum will be enough to get started, I leave it at that.

Next is the region where the machine will live.

Since most of you are in the US, I will just leave it in SF.

It also gives me the choice of password and SSH based authentication. As always with more security comes less convenience. Not sure how inconvenient it will be, but I guess I will somehow get along with SSH, so lets do that…

Im vaguely familiar with the process since Im using SSH to log in to a lot of my local machines via terminal, so I fumble my way through, generate a key pair, and enter the public key into the respective field.

However, it throws some generic error “The backend responded with an error”,

and some googling didnt help much but brought up a few reports from users saying they have seen that error and it disappeared a few minutes later.

So I will leave it at that for now and do some other duties.

A few minutes later, it still throws the same error. I google some more, and come across this thread:
https://news.ycombinator.com/item?id=20064169

Not that it would contribute to overcoming the error I experience, but it makes me think that I should consider an automated external backup of the forum content once moved to DO. Currently, all eggs are in one basket, at discourse. If they would screw up and our data gets lost there somehow, I have no backup. FYI, the complete forum instance is currently using just over 10GB.

An hour later, still same stupid error message.

For now, Im enabling the DO backup feature for an extra buck a month. Note: this will save the complete instance, so it depends on the DO infrastructure for access and execution. This will not replace a scheduled discourse export.

and I select “one-time-password” to move past the SSH error message.

alright, after setting it to OTP, the “Create Droplet” button turned green, and after clicking it, it threw a whole bunch of not helpful error message, and no droplet was created.

I started again, clicked through all the option quickly, and voila things fell in place:

Moving on :smiley:

I log onto the created machine via SSH, and are greeting with the following:

Oops, it complains about not having enough RAM. Yeah I remember ignoring that earlier… So lets go back and resize:

and then restart the installer.

Script running through, wants a domain. Says it cannot run with just an IP.
So I create an A record in my DNS settings that redirects community-dev.micro-motor-warehouse.com to the IP of the VM.

restart the installer.

It still fails with the same error:

I assume it will take some time to percolate, so I will leave it for now and do some other duties.

Hey awesome progress!
i do however think its complaining rightly so. A dig on community-dev.micro-motor-warehouse.com
Only gives me a CNAME to the shopify storefront.

dig A community-dev.micro-motor-warehouse.com.micro-motor-warehouse.com
on the other hand gives me the proper A record.
community-dev.micro-motor-warehouse.com.micro-motor-warehouse.com. 21599 IN A 157.245.165.64

A and AAAA records propagate pretty quickly these days. you can expect them to hit your local DNS within 10 minutes.

thanks mate!
When I go to shorthen this entry


to this:

it gives me this error message:

What do you suggest?

See if there is already an record with that name. Could be of different type (like CNAME?) if shopify is trying to be picky.

1 Like

Indeed, me stumbling around in the config added a useless CNAME.
Got rid of it, things are moving again. Sweet :smiley:

1 Like

Well, not so quick…
I have added community-dev A record as per above, no error message while adding it, sits there now.
However upon running the script 10 minutes later, still same error message that the server is not accessible on that URL on 443 and on 80.

Need to leave now. Will continue in a few hours.
If you are still around, please do some more digging :wink:

For me it reports the correct ip now. Maybe just try again when you get back?

Alternatively, dig is literally a linux command line program to lookup DNS records, try running
dig community-dev.micro-motor-warehouse.com
via ssh to see if the server has gotten the update too yet.

If it complains that it can find dig, install it via sudo apt-get install dnsutils

alright, it just took time.
I just restarted the script, and it gets past that step now, and I see a lot of compiler messages.
So thats good news.
Leaving it alone for now, getting other stuff done.
Later!

1 Like

yeah buddy!
thanks, @bkleiner! You have been instrumental to get to this point.

Next: configure the forum software and migrate the content.

3 Likes

well, that was quite easy. Backup and restore took just a few clicks.
I have imported the complete content to

You may go over there and play around, with the same username and password.
Email notifications are not enabled yet.
My hosting plan at discourse just renewed a few days ago, so we have another few weeks paid here.
At the time of the final move, I will redirect the community. subdomain to the new server, and hopefully most users wont even notice it.
Until then, as long as the new server can be reached under community-dev. subdomain, you are able to post over there, but those posts will eventually be deleted, as I will do another backup and restore at the same time when I redirect the domain.

Anyone who want to check out if everything works on the new page, or has any suggestions what I should consider for the move, let me know!

8 Likes

Awesome! if you need any help in the future, feel free to hit me up.

3 Likes

Fantastic Job @Benedikt & Benedikt #2 @bkleiner

Not trying to pressure you for a response at all, but I do not want you to think that I do not care about your personal situation and I know that you have your hands full with life - I did send you an email via MMW customer support

Yeah I tend to get that after a bad curry :face_vomiting:

I know, I am childish but it’s part of my “charm” :slight_smile:

Great to see this happening @Benedikt

1 Like

To be totally honest with you, every single thing you have done so far has been absolutely totally over my head. So, I’d like to say THANK YOU, to you guys for making it so. You guys ROCK and I, for one, appreciate You.

3 Likes