This is an old story, perhaps a year ago, when I was working at one of telecommunication vendor. And I thank God, I got a lot of lessons and experiences while working there.
One day, my friend from NOC told me that the network couldn’t monitored from Cacti last night, but there was no traffic down. He asked me to check what happened.
Well, I was thinking that it was NMS problem. I started to check the alarm log on IPRAN core device.
An alarm 400136 ID 180933 level 5 occurred at 04:40:23 04-26-2016, cleared at 05
:54:05 04-26-2016 sent by xxx_9K8E_xxx_1461 PFU-0/1/0
%PORT% Physical interface status The physical interface xlgei-0/1/1/1 turned in
From the log I got notice that port xlgei-0/1/1/1 was down before, I found that this port is main link to IPBB PE router. So definitely the traffic passed through the backup link.
But wait, why no traffic shown by Cacti server?
Checked the routes configuration :
xxx_9K8E_xxx_1461#show running-config static
ip route vrf NMS 10.17.11.45 255.255.255.255 172.28.106.1 name NTP
ip route vrf NMS 10.17.11.45 255.255.255.255 172.28.106.13 100 name NTP
ip route vrf NMS 10.17.71.0 255.255.255.0 172.28.106.1 name VPN
ip route vrf NMS 10.17.120.0 255.255.255.0 172.28.106.1 name CACTI
Well, only 1 route toward Cacti server, through next-hop IP 172.28.106.1.This next-hop IP is IP on IPBB PE router. Since P2P connections use /30, the IP address of IPRAN core is 172.28.106.2.
xxx_9K8E_xxx_1461#show ip interface brief | i 172.28.106.2
xgei-0/1/0/22.00000032 172.28.106.29 255.255.255.252 up up up
xlgei-0/1/1/1.00000401 172.28.106.2 255.255.255.252 up up up
xgei-0/4/1/22.00000032 172.28.106.25 255.255.255.252 up up up
From the result above, 172.28.106.2 belongs to interface xlgei-0/1/1/1, and this is the main link.
So why the traffic couldn’t monitored by the Cacti server when the main link down? Yes, because there was no backup route to Cacti server through the backup link.