Last updated: 01/07/2019
After a prolonged outage, our Nextcloud server is now back up and running! This upgrade saw a complete overhaul of the server including an upgrade to Debian 9.5 and PHP7 as well as Nextcloud 15. In addition, a new Raid controller was installed to manage the mirrored SSD boot disks and new Raid6 configuration of the main storage drives. This new configuration allows for greater redundancy with the ability to loose two drives and continue operating. The addition of the raid card over the older software raid also saw an increase in read/write performance as well, helping with the offload of other tasks from the server. Plex, and the main webserver now reside on a separate virtual machine in an effort to tighten security by reducing the number of attack vectors on the server. Operation should now be normal with no more plans for extended downtime. Focus now goes towards adding addition services that take advantage of the SQL server running on the storage server such as an RSS reader, webmail, and integrated document editing in Nextcloud. The backup situation remains unchanged, with local hourly backups of the live data and nightly backups to sites in California and Illinois.
We are also using a new Internet Service Provider in our office. Alyrica Networks, a WISP local to Corvallis now provides Internet connectivity at a speed of 6/6Mbps with the option to upgrade if we see the need. We have also managed to purchase an HP DL380p Gen8 server that will soon be taking over as the ESXI virtual machine host from our older Sun x4150. This will hopefully see lower server room temperatures and increased Plex performance allowing for more simultaneous connections.
UPDATE: Document editing has been fixed as of this time. It seems a problem with configuration files being kept while upgrading the document editing service broke things, with this seeming to be a known issue (multiple recent Github issues created.) Purging and reinstallation was necessary to fix the problem.
Monday afternoon I became aware of the fact that document editing was no longer working on the cloud server. After some investigation I found that the document editing service, which sits on another server was not functioning correctly. CPU thrashing was observed, and I am currently in the process of reinstalling the document editing service. There is no ETA as to when document editing will be available again, but no data loss has, or is expected to occur.
After a short outage of one day, everything is back up and running. While this long of an outage was not planned, downtime due to maintenance was expected as I brought the new router for EASC online. This new router takes over the role of the Comcast gateway, which had been struggling for a long time. The new router is a dedicated box running PFsense that is capable of many more things than the previous gateway, including VPN server, VLANs, more in depth firewalls, and reporting of more information. This implementation has also allowed me to segment off the wireless network of EASC's office from the wired network, further securing computers and devices run by EASC.
Created and maintained by Drake Vidkjer, for more information contact me at drake@crvit.com