The Fora: A Higher Education Community

General Category => General Discussion => Topic started by: eigen on December 04, 2019, 12:32:44 PM

Title: Brief Forum Downtime
Post by: eigen on December 04, 2019, 12:32:44 PM
Hi all-

Sorry for the downtime this morning. There were some processes that got stuck on the forum server, and were causing it to run out of memory. They're cleared and we're back up!

I'll keep an eye on things to make sure they don't creep back.

Cheers,

Eigen
Title: Re: Brief Forum Downtime
Post by: mamselle on December 04, 2019, 12:50:05 PM
Quote from: eigen on December 04, 2019, 12:32:44 PM
Hi all-

Sorry for the downtime this morning. There were some processes that got stuck on the forum server, and were causing it to run out of memory. They're cleared and we're back up!

I'll keep an eye on things to make sure they don't creep back.

Cheers,

Eigen

Thanks for all you do!

M.
Title: Re: Brief Forum Downtime
Post by: namazu on December 04, 2019, 01:16:15 PM
Thanks, Eigen!
Title: Re: Brief Forum Downtime
Post by: ab_grp on December 04, 2019, 02:18:27 PM
Adding my thanks for all your work on this site.  So, thanks!
Title: Re: Brief Forum Downtime
Post by: Parasaurolophus on December 04, 2019, 09:32:41 PM
Quote from: eigen on December 04, 2019, 12:32:44 PM

Sorry for the downtime this morning.

I might maybe be willing to forgive you at some point in the future...
Title: Re: Brief Forum Downtime
Post by: Hibush on December 05, 2019, 06:38:49 AM
Thanks for finding and fixing that issue!
Title: Re: Brief Forum Downtime
Post by: EdnaMode on December 05, 2019, 06:47:13 AM
Thanks for all you do! I had a bit of a moment (not panic, nope, of course not) when I saw that it was down yesterday :)
Title: Re: Brief Forum Downtime
Post by: downer on December 05, 2019, 07:38:14 AM
Thanks Eigen

The thought occurred to me that it might be helpful to have somewhere else to post enquiries about the status of The Fora when it is down. Somewhere like a Facebook group, or even Twitter. But it might be more trouble than it is worth.
Title: Re: Brief Forum Downtime
Post by: Thursday's_Child on December 05, 2019, 07:58:54 AM
While composing my previous post it froze several times and I got a message that thefora.org wasn't responding.  Could this be related?  It doesn't usually happen.
Title: Re: Brief Forum Downtime
Post by: mamselle on December 05, 2019, 09:16:00 AM
Quote from: downer on December 05, 2019, 07:38:14 AM
Thanks Eigen

The thought occurred to me that it might be helpful to have somewhere else to post enquiries about the status of The Fora when it is down. Somewhere like a Facebook group, or even Twitter. But it might be more trouble than it is worth.

Yes, because when I tried to use the email address suggested as an alert, it was apparently a part of the problem, so could not be a part of the solution....or at least, I got a "non-functioning email" reply a few moments later...

M.
Title: Re: Brief Forum Downtime
Post by: eigen on December 12, 2019, 03:50:44 PM
So an update on downtime (some of you might have noticed we had another outage a few nights ago):

It looks like we've had a *sharp* uptick in "guests", many of who are bots. This is putting an inordinate load on the servers in terms of number of parallel processes running.

I've put some more severe limits on what guests can view to see if that helps. If you're used to browsing as a guest, you may need to start logging in.

If you notice detrimental downsides, let me know!
Title: Re: Brief Forum Downtime
Post by: traductio on December 13, 2019, 09:39:07 AM
Quote from: eigen on December 12, 2019, 03:50:44 PM
So an update on downtime (some of you might have noticed we had another outage a few nights ago):

It looks like we've had a *sharp* uptick in "guests", many of who are bots. This is putting an inordinate load on the servers in terms of number of parallel processes running.

I've put some more severe limits on what guests can view to see if that helps. If you're used to browsing as a guest, you may need to start logging in.

If you notice detrimental downsides, let me know!

Are you at all concerned about the effect of this change on lurkers? I know it took me five minutes to find this explanation, and I've been on the Fora since the CHE days. But I was also a lurker for quite a long time, and like many, I made good use of the advice I could find easily and for free. Creating an account would have been enough of an obstacle that I wouldn't have bothered.
Title: Re: Brief Forum Downtime
Post by: eigen on December 13, 2019, 10:11:30 AM
Quote from: traductio on December 13, 2019, 09:39:07 AM
Quote from: eigen on December 12, 2019, 03:50:44 PM
So an update on downtime (some of you might have noticed we had another outage a few nights ago):

It looks like we've had a *sharp* uptick in "guests", many of who are bots. This is putting an inordinate load on the servers in terms of number of parallel processes running.

I've put some more severe limits on what guests can view to see if that helps. If you're used to browsing as a guest, you may need to start logging in.

If you notice detrimental downsides, let me know!

Are you at all concerned about the effect of this change on lurkers? I know it took me five minutes to find this explanation, and I've been on the Fora since the CHE days. But I was also a lurker for quite a long time, and like many, I made good use of the advice I could find easily and for free. Creating an account would have been enough of an obstacle that I wouldn't have bothered.

Having the forum crash constantly due to the guest traffic will also have a negative effect on lurkers.

There's no way to differentiate lurkers who are legitimate users from the ~50 or so constant "guests" who are bots, and every page they are able to access adds significantly to the load on the servers.
Title: Re: Brief Forum Downtime
Post by: Cheerful on December 13, 2019, 11:42:59 AM
Too bad.  Some lurk for years to gain info and don't want to register.  I accessed The Fora as Guest repeatedly today and thought it was down.  I've lurked on other boards for years and no such problems there.  Guess it depends on server issues.

Good to know many recent "guests" are bots.  I naively thought those stats indicated increasing interest. : (

Thanks for all you do to keep The Fora going. : )
Title: Re: Brief Forum Downtime
Post by: Liquidambar on December 13, 2019, 01:15:29 PM
If displaying this number of boards has the desired outcome on server load, maybe we can give some thought to which board would draw in the most new members?  The job board seemed to be a big draw to the old fora, so I'd be inclined to vote for that one.
Title: Re: Brief Forum Downtime
Post by: eigen on December 13, 2019, 08:18:54 PM
Quote from: Liquidambar on December 13, 2019, 01:15:29 PM
If displaying this number of boards has the desired outcome on server load, maybe we can give some thought to which board would draw in the most new members?  The job board seemed to be a big draw to the old fora, so I'd be inclined to vote for that one.

I think that's a great idea. Lets see how the servers handle things over the next few weeks, and whether it makes a huge difference or not.

Please keep suggestions coming if you have them!
Title: Re: Brief Forum Downtime
Post by: dr_codex on December 13, 2019, 08:36:39 PM
Quote from: eigen on December 13, 2019, 08:18:54 PM
Quote from: Liquidambar on December 13, 2019, 01:15:29 PM
If displaying this number of boards has the desired outcome on server load, maybe we can give some thought to which board would draw in the most new members?  The job board seemed to be a big draw to the old fora, so I'd be inclined to vote for that one.

I think that's a great idea. Lets see how the servers handle things over the next few weeks, and whether it makes a huge difference or not.

Please keep suggestions coming if you have them!

No suggestions, because I don't understand the technical issues, but another Thanks! for keeping this running.

I'd like to keep lurkers lurking -- I did this for many years on the CHE version -- but if the bots are coming, and this isn't possible, would there be a way to nudge actually curious people on a welcome page? E.g.: Because of bots, you'll have a better experience if you create an account. Accounts are pseudonymous, and you can reboot at any time.

Phrased more elegantly.

Thanks mods all.
dc
Title: Re: Brief Forum Downtime
Post by: traductio on December 14, 2019, 08:09:25 PM
I, too, am grateful to all who have worked to continue the Fora in its new form. I just did the mental math -- I started reading the CHE fora in 2006, so 13 years ago. (I'm at the point in my career where I do lots of mental math and realize that I've been doing X or Y for a decade and a half. It's happened, folks -- I've become my parents.) I also just looked at the rate at which I post -- about .24 posts per day. It's been that way since I first registered in 2008!

As for letting lurkers know there's more than meets the eye -- perhaps a pinned post on the General Discussion board?
Title: Re: Brief Forum Downtime
Post by: namazu on December 14, 2019, 08:26:46 PM
What's the current hosting arrangement, and what would it take to upgrade the servers so that they could handle 50 bots and a modest legion of lurkers?  Is there a feasible/reasonable way to do that? (If money is a limiting factor, I suspect some of us would be willing to chip in to help keep the lights on.)
Title: Re: Brief Forum Downtime
Post by: eigen on December 14, 2019, 10:17:58 PM
Quote from: namazu on December 14, 2019, 08:26:46 PM
What's the current hosting arrangement, and what would it take to upgrade the servers so that they could handle 50 bots and a modest legion of lurkers?  Is there a feasible/reasonable way to do that? (If money is a limiting factor, I suspect some of us would be willing to chip in to help keep the lights on.)

It's hard to get a number on that, but my guess is it's going to be a significant change.

We're on shared hosting donated by a company right now (Reclaim Hosting) to see if this will work and how our needs grow/expand/change.

There's definitely a possibility that we'll need to move to dedicated rather than shared hosting or to a beefier host at some point in the future, and we'll have see what our options are as we assess this.