Outage - PennREN Various Services via Core Node NBRD ISSUE=3131 PROJ=131

Wed Oct 22 19:16:37 UTC 2014

Just an FYI
We have shut down our peer. Please let us know when we can re-enable it

Here are our logs if it helps troubleshooting your problems.

Oct 22 14:31:41: %BGP-3-NOTIFICATION: received from neighbor 162.223.16.230 4/0 (hold time expired) 0 bytes 
Oct 22 14:31:41: %BGP-5-ADJCHANGE: neighbor 162.223.16.230 Down Peer closed the session
Oct 22 14:31:41: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  Peer closed the session
Oct 22 14:34:53: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 14:34:53: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 14:34:53: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 14:35:34: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 14:36:51: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 14:38:08: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 14:38:08: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 14:38:08: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 14:38:18: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 14:39:30: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 14:41:20: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 14:41:20: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 14:41:20: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 14:42:45: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 14:44:34: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 14:44:34: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 14:44:34: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 14:45:58: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 14:47:46: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 14:47:46: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 14:47:46: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 14:49:11: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 14:50:55: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 14:50:55: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 14:50:55: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 14:52:24: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 14:54:02: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 14:54:02: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 14:54:02: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 14:55:32: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 14:57:17: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 14:57:17: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 14:57:17: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 14:58:40: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 15:00:28: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 15:00:28: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 15:00:28: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 15:01:54: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 15:03:37: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 15:03:37: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 15:03:37: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 15:05:06: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 15:06:44: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 15:06:44: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 15:06:44: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 15:08:13: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 15:09:55: %BGP-3-BGP_NO_REMOTE_READ: 162.223.16.230 connection timed out - has not accepted a message from us for 180000ms (hold time), 0 messages pending transmition.
Oct 22 15:09:55: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 active 4/0 (hold time expired) 0 bytes 
Oct 22 15:09:55: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  BGP Notification sent
Oct 22 15:11:22: %BGP-3-NOTIFICATION: sent to neighbor 162.223.16.230 passive 2/8 (no supported AFI/SAFI) 3 bytes 000000 (timer expired)
Oct 22 15:11:50: %BGP_SESSION-5-ADJCHANGE: neighbor 162.223.16.230 IPv4 Unicast topology base removed from session  Admin. shutdown
Oct 22 15:11:51: %SYS-5-CONFIG_I: Configured from console by romeo on vty0 (192.168.30.131)

Romeo Czumbil
Sr. Network Engineer

1000 Adams Ave | Norristown, PA 19403
tel 610-994-3046 | cell 609-220-0322
rczumbil at xand.com

website  |  news  |  blog  |  data centers   

Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

-----Original Message-----
From: pennren-ops-l-bounces at pennren.net [mailto:pennren-ops-l-bounces at pennren.net] On Behalf Of PennREN Network Operations Center
Sent: Wednesday, October 22, 2014 2:52 PM
To: pennren-ops-l at pennren.net
Subject: Outage - PennREN Various Services via Core Node NBRD ISSUE=3131 PROJ=131

SUBJECT:               Outage - PennREN Various Services via Core Node NBRD
AFFECTED:              Core Node NBRD
STATUS:                Unavailable
START TIME:            Wednesday, October 22, 2014, 6:31 PM (1831) UTC
END TIME:              ***PENDING
DESCRIPTION:           Services from Core Node NBRD were briefly unavailable. PennREN Engineering is
                       currently diagnosing the root cause of the outage. Service has rerouted.
                       Although some traffic flows may have been impacted initially, traffic was
                       automatically rerouted around the outage.
TICKET NO.:            3131:131
TIMESTAMP:             Wed Oct 22 18:52:04 2014 UTC

Message ID:            PREN.3131.1.1

The times provided above are in Coordinated Universal Time (UTC).

When replying, type your text above this line.
Please submit problems, requests, and questions at:
http://globalnoc.iu.edu/pennren/support/report-a-problem.html

Thank You,
PennREN Network Operations Center
noc at pennren.net, 317-274-3213

PennREN NOC Home Page: http://noc.pennren.net     
PennREN NOC Operations Calendars (RSS and ICAL):
http://globalnoc.iu.edu/pennren/support/operations-calendar/pennren-calendars.html