Realtime logging monitor from multiple ec2 instances on AWS
When you have to manage multiple ec2 instances (or multiple servers in general), you have many log files to monitor, and usually the situation is to have simultaneous ssh connections and all is very hard to manage.
I would like to describe my solution to this problem, the goal is to have a simple web interface to monitor all files.
We are going to use this services:
- multiples ec2
- log.io free software (http://logio.org)
- internal DNS (Route 53)
- custom amazon security group
Ok… Let’s go!
First of all, imagine to have this scenario:
- SERVERA (web server) internal ip 172.0.0.1
- SERVERB (mail server) internal ip 172.0.0.2
- SERVERC (nodejs application server) internal ip 172.0.0.3
- SERVERD (log server) internal ip 172.0.0.4
The idea is to have a log server (SERVERD) that receive logs from all others machines and serve it through a web interface to all clients. Obviously we want to use SSL connection and HTTP authentication to view logs.
STEP1 — Create a local dns on Route 53
To easily manage all the machines, we can create a local DNS in our VPC, we call it “locadomain.com”. From aws console panel, go to “Route 53” service, choose “Hosted zone” and create a new zone:
In the form below fill all the fields, and choose a VPC for your DNS:
As described in the alert box, remember to set “enableDnsHostnames” and “enableDnsSupport” to true, you find this options in the “VPC” section of console (right click in your “VPC”).
Inside the hosted zone, create a record for each ec2, choose as type “A — IPV4 Address” and for name “SERVERA, SERVERB, ecc…” in the value field insert the “internal ip address” of each instance.
To make sure that all works fine, ssh in each instance and test the dns with the command:
host servera.localdomain.com
if you see the correct ip in response, you have done!
STEP2— Install and configure log.io software
You can install this software via npm:
sudo npm install -g log.io — user “ubuntu”
You have to install this software on all instances.
Inside SERVERD (log server) edit file: “web_server.conf”
sudo nano /home/ubuntu/.log.io/web_server.conf
uncomment the sections:
// Enable HTTP Basic Authentication
auth: {
user: “admin”,
pass: “12345”
},// Enable HTTPS/SSL
ssl: {
key: ‘/path/key.pem’,
cert: ‘/path/cert.crt’
},
Save the file and exit. and launch nodeserver:
sudo log.io-server
In all others servers (A,B,C) edit the file: “harvester.conf”
sudo nano /home/ubuntu/.log.io/harvester.conf
and edit the last section like this:
server: {
host: ‘serverD.localdomain.com’,
port: 28777
}
and the section called “logStreams” with the files you want to monit on each instance:
apache: [
“/var/log/apache2/access.log”,
“/var/log/apache2/error.log”
],
myapplication: [
“/path/mylog.log”
]
Save the file and exit. and launch harvester:
sudo log.io-harvester
Now create a security group in your amazon console called “monit” and open this ports: 28778 for all ips, and 28777 only for your VPC. If you do not want SSL do not uncomment the last part.
Assign this security group to all your instances.
STEP3 — Monit your files
When you have done, go to public IP of your SERVERD:
http://SERVERD_PUBLIC_IP:28778
insert user “admin” and password “12345” and enjoy! this is the result:
Clap this article if you like it! thanks!
You may also like this stories:
