The Digital Ocean droplet housing Tux Reports Network was hacked the other day. I was not able to work on things until yesterday morning and this gave me time to (re)think about the best strategy to recover from a hacker.
I decided that instead of simply restoring an image (snapshot of the server) then looking for the security hole I would create a new droplet with different features. Specifically the new droplet does not include nginx or varnish.
After 104,000 files were uploaded, along with 6 databases, it was time to tune the server.
Top and the Digital Ocean control panel graphs showed spikes happening every 30 minutes. Here is a view of running top, showing that the %CPU is too high, and tuning is required.
Editing my.cnf
The script mysqltuner was installed and run several times to adjust my.cnf file sitting in /etc/mysql directory.
Next, Matthew Montgomery’s MySQL performance tuning primer was helpful in editing the my.cnf.
The following graph shows the decrease in spiking after adjusting key_buffer_size, table_cache, and max_heap_table_size.
Of course, the server will need to run at least 48 hours longer before I can run the script to actually optimize the my.cnf but at least the spikes are down.
Benefit #1
The 503 errors appear to no longer be an issue. I’m sure it was an incorrect configuration in the older droplet but it is nice to not have this issue.
Benefit #2
An interesting aside is that the Moodle installations appear to run better than under the previous droplet. PDF files are still an issue and do not open properly. The moodle course data is being reuploaded in an attempt to see if the files were corrupt.
Update: Oops. A backup script ran and overpowered the server.