Servers, CI, and you!

[John Bailey]

On 10/03/2012 05:47 AM, Evan Kinney wrote:
> I'd like to possibly revamp eider and duck.

With the cPanel junk you note below, it would probably be easier if NetworkRedux
could provide you two brand new VM's to be properly configured.

> They're both running long-in-the-tooth versions of CentOS 5 and, to be
> quite honest, no one is really sure of CentOS's future given the
> nature of the project and the fact that Red Hat keeps making it more
> difficult to build their SRPMs into something usable. In my opinion,
> this leaves us with two options: the Ubuntu LTS spin (or Debian
> Stable, I suppose) or Fedora.
> 
> I've always been a Red Hat guy and I basically know RHEL (and, thus,
> to some degree Fedora) like <insert something I know everything about
> here>, so my natural inclination is to go with Fedora, but I have some
> reservations with hosting a public-facing server on a platform that
> releases so often and stops supporting releases after 13 or so months.
> Ubuntu LTS sounds like a pretty good option to me. Thoughts, anyone? I
> suppose we're also limited by the base images that Network Redux
> provides for their OpenVZ instances.

In my opinion, using Fedora on a server is highly irresponsible.  As you
mention, each release gets only 13 months of support, and there are often new
releases of software added in; in a server environment this is far from ideal.
On a server you generally want things not to change very much with updates until
a new OS release happens.  RHEL/CentOS is better about this.

Speaking from experience, we've (Pidgin) been mostly happy with our Debian
Stable-based VM's (our main complaints are that they're OpenVZ VM's instead of
something sane like Xen or real hardware).  There's not much difference between
the two, except that Ubuntu, in my experience, tends to patch packages more than
Debian does.  That said, however, those of us who do the administration work for
Pidgin's servers are far more comfortable on a Debian box than on a RedHat-style
box.

> As part of this, I'd like to propose we get rid of all the cPanel
> cruft that's currently holding up everything on duck. I've never
> really been a fan of cPanel (their installer, for instance, is a shell
> script they suggest you pipe to bash via cURL that essentially
> modifies your system to the point of no return) and, as far as I can
> tell, there's nothing we're doing that requires it.

cPanel should die ASAP.  All it ever does is get in the way of people who know
what they're doing.

> Also, it looks like duck and eider are two OpenVZ VMs in the same
> Network Redux datacenter. duck has 3 cores (any reason for 3?) with
> 6GB of RAM, and eider has 2 cores with 2GB of RAM. What if we were to
> combine those together and just have one larger VM? As long as
> everything's properly configured (and given the way things are
> currently set up), I can't think of any reason to have two separate
> machines. Another alternative would be to split them equally, cluster
> them with a pacemaker/corosync stack, and load balance everything with
> the help of HAproxy. I have a lot of experience doing that, but I know
> it's not exactly the easiest thing to maintain... so maybe simple is
> better here, even if we're giving up high availability.

One other option you might want to consider is having one VM be nothing but a
database server, and the other VM be the frontend stuff (trac, the xtras site,
etc).  NetworkRedux can give you a private VLAN to isolate this traffic and keep
it off the public network.  In some cases this can make trac significantly faster.

> We might also want to look at cleaning up the DNS zones a bit, as
> they're a bit of a mess if the current Apache configs are any
> indication. What if we had everything use .adium.im, and had all of
> the .adiumx.com URLs redirect there instead of serving the content?
> This would also make it easier to manage SSL... which is another thing
> I plan on making work properly (and has been discussed on here
> before).

Redirects are cheap enough that I have to agree with you here, but I have no
knowledge of how all the stuff you guys run works, so take that with a grain of
salt.

> I know this is a lot, but it's stuff I think needs to be done at some
> point. Everything would be a *lot* more maintainable, more secure, and
> (most likely) significantly faster (especially the Mercurial web
> interface). I'm willing to make all this happen, but I'd like to hear
> some input and discussion before I put together a formal proposal for
> consideration.

There are a couple things I'd like to discuss with you off-list about monitoring
the servers.

John

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://adium.im/pipermail/devel_adium.im/attachments/20121003/39df8ed1/attachment.sig>