Don’t Panic

This is an old story, perhaps a year ago, when I was working at one of telecommunication vendor. And I thank God, I got a lot of lessons and experiences while working there.

One day, my friend from NOC told me that the network couldn’t monitored from Cacti last night, but there was no traffic down. He asked me to check  what happened.

Well, I was thinking that it was NMS problem. I started to check the alarm log on IPRAN core device.

An alarm 400136 ID 180933 level 5 occurred at 04:40:23 04-26-2016, cleared at 05

:54:05 04-26-2016 sent by xxx_9K8E_xxx_1461 PFU-0/1/0

%PORT% Physical interface status  The physical interface xlgei-0/1/1/1 turned in

to DOWN

From the log I got notice that port xlgei-0/1/1/1 was down before, I found that this port is main link to IPBB PE router. So definitely the traffic passed through the backup link.

But wait, why no traffic shown by Cacti server?

Checked the routes configuration :

xxx_9K8E_xxx_1461#show running-config static

!<static>

..<-output omitted->..

ip route vrf NMS 10.17.11.45 255.255.255.255 172.28.106.1 name NTP

ip route vrf NMS 10.17.11.45 255.255.255.255 172.28.106.13 100 name NTP

ip route vrf NMS 10.17.71.0 255.255.255.0 172.28.106.1 name VPN

ip route vrf NMS 10.17.120.0 255.255.255.0 172.28.106.1 name CACTI

..<-output omitted->..

Well, only 1 route toward Cacti server, through next-hop IP 172.28.106.1.This next-hop IP is IP on IPBB PE router. Since P2P connections use /30, the IP address of IPRAN core is 172.28.106.2.

xxx_9K8E_xxx_1461#show ip interface brief | i 172.28.106.2

xgei-0/1/0/22.00000032          172.28.106.29   255.255.255.252   up    up   up

xlgei-0/1/1/1.00000401           172.28.106.2     255.255.255.252   up    up   up

xgei-0/4/1/22.00000032          172.28.106.25   255.255.255.252   up    up   up

From the result above, 172.28.106.2 belongs to interface xlgei-0/1/1/1, and this is the main link.

So why the traffic couldn’t monitored by the Cacti server when the main link down? Yes, because there was no backup route to Cacti server through the backup link.

 

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s