Verizon Business support is really awful.

So we've got a Verizon supplied internet connection.

Given a number to call for BGP related things so called this number, put it on speaker-phone as i waited and was stuck on hold for 40 minutes before i gave up, with no opportunity to request a call back... Just apology messages and hold music... /me gets tad frustrated

I know what i'll do ... I'll log a trouble ticket and hopefully that'll poke them into giving me a call back. So trouble ticket gets logged, and im promised a call back within 4 hours, I'm advised that if 4 hours passes to give them a call back.

Guess what? So over 4 hours passes . I call back verizon support, politely inquiring whats happened, and why i havent been contacted... Advised that a tech is working on the case currently, just hasnt contacted me.. So i request call is transferred to tech.. Guess what... Hes not available. Person at other end of the phone says he doesnt get into the office for another 5 hours!! AT this i really do have to laugh.

So i request that case is transferred to another tech, one who is in the office... Support goes OK. Support when can i expect a call back... It'll be another 4 hours..

/me sighs... All im trying to do is get somebody to flick a switch... Ask support, any chance of a sooner callback ? Line gets cut off...

So thats over an hour wasted on the phone to verizon today, with no sense of whats being done about the ticket BGP peering still not sorted. And no idea of when im getting a callback... And if i decide to make the investment of making another call to support, it takes a minimum of 2minutes and 12 seconds to listen to all their advertising and mash the keypad enough to get through to an actual person...

Verizon, by quite a large margin, you truly do have the worst support of any vendor I have the pleasure of working with.


Rant over.....

Cisco Archive config.

Setting up a new infrastructure and their switches have an archival config like this, meaning that the config is written to the ftp path nominated every time a user issues the "wr mem" command and otherwise every 12 hours if there are changes. The "path" uses 2 variables namely $h == hostname of swtich and $t == time of archive upload.

archive
 log config
  logging enable
  notify syslog contenttype plaintext
  hidekeys
 path ftp://windows-ftp-server/$h-$t
 write-memory
 time-period 720

Noticed however that this was not working on 2 out of 3 switches... when i looked into it it turned out to be only an issue when the $t variable was used.

After a quick pcap of what was going on  (cisco's logs didnt give any info about this) i found the format of the time specified by $t on 2 of the swtiches was like this "Aug-27-07:37:06.147" while on the other it was in the format "Aug-27-12-01-16.548"... This was preventing the upload from working on 2 switches as there wree colons : in the file name, which arent a valid character for a windows filename. Given a windows FTP server was in use the upload failed. :(

I noticed that this was an issue on :
- c3750-advipservicesk9-mz.122-44.SE.bin
- c3750e-universalk9-mz.122-50.SE3.bin

But not on:
- c3750-ipservicesk9-mz.122-52.SE.bin

Upgraded all switches to 12.2(52) and things are looking better now.

Manage Cisco ASA over vpn connection

ARGH.. this drove me mad.. A surprisingly simple / common thing people may want to do is
- setup a Cisco ASA device (in this case a 5505) at a satellite office
- establish an VPN over the internet from this device to the main office
- manage device remotely via this VPN link.

In this case all that was required, in addition to the vpn setup was:
ssh 0.0.0.0 0.0.0.0 inside
ssh timeout 30
ssh version 2
management-access inside

The key being "management-access inside", just thought i'd post about it because it drove me mad, and its so easy when you know how.

Heres a link:
https://www.cisco.com/en/US/docs/security/pix/pix63/command/reference/mr.html#wp1137951

Cisco IOS and ASA Packet Captures

Sometimes getting a packet capture at a network pinch point can be useful.. where i work, this normally needs happen for debugging purposes on a firewall, and sometimes on IOS based routers....

Heres how i do it on IOS:

! in enable mode 

! creates a circular buffer of up to 1024K

 monitor capture buffer mybuffer size 1024 max-size 1024 circular

! Creates a capture point in this case subinterface 100 of Gig 0/1

 monitor capture point ip cef mycapture gigabitethernet0/1.100 both

! Associte the buffer with the capture point

monitor capture point associate mycapture mybuffer

! Start the capture

monitor capture point start mycapture

! Stop the capture

monitor capture point stop mycapture

! show the capture (not particularly useful unless you can read hex)

show monitor cap buffer mybuffer dump

! often better to upload the capture to somewhere and use wireshark

monitor capture buffer mybuffer export tftp://10.0.0.1/mycapture.pcap

And on ASA
! this ACL can be used to filter what is captured

access-list captureacl permit ip any any

! This sets up the capture on the inside interface, to use a specific ACL.

! buffer size in bytes as well as circular-buffer are set
capture mycapture access-list captureacl interface inside buffer 100000 circular buffer

! Upload capture.

copy capture:mycapture tftp://10.0.0.1/mycapture.pcap

DNS replies being dropped when traversing my ASA.

So we came up agains a problem recently where some DNS replies were getting dropped when traversing my Cisco ASA firewall. But why?
So to check this out we did a few packet traces while executing dns queries (using nslookup) which both worked and did not work. It quickly became apparent that it was replies of over 512 bytes that were having problems... But why ?!

So i reviewed my firewall configs and for the ruleset in question there was nothing defined in my access-lists which was denying this traffic, for the particular connection we were allowing ip any any to traverse the firewall. So then i looked at my service-policy and compared it to the cisco best practis doc here:
http://www.cisco.com/web/about/security/intelligence/dns-bcp.html

Everything checked out, this was the config:
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum 512

Given the packet sizes we had already seen, i was certain that my problem was directly related to the message-length 512 parameter. I just didnt want to go change it without understanding. So i read (scanned through) these RFC's:

• http://www.ietf.org/rfc/rfc1035.txt (relating to the original DNS spec, seems to confirm 512byte message size is correct, suggesting the Cisco Best Practise is correct also)

• http://www.ietf.org/rfc/rfc2671.txt (relating to EDNS, ahhh so the message lenght between EDNS compatible client / servers can be longer.. MUCH longer)

Confirmed this by trying out some dig's...
~> dig +short rs.dns-oarc.net txt

;; connection timed out; no servers could be reached

Bit more research suggested that my bind DNS install by default advertises a edns-udp-length of 4096 ... http://www.zytrax.com/books/dns/ch7/hkpng.html , didnt know that! But at least now its all falling into place.

Updated my FW policy to this knowing i needed a 4K message-length.
policy-map type inspect dns preset_dns_map

 parameters
  message-length maximum 4096

And now it all works:
~>dig +short rs.dns-oarc.net txt
rst.x3827.rs.dns-oarc.net.
rst.x3837.x3827.rs.dns-oarc.net.
rst.x3843.x3837.x3827.rs.dns-oarc.net.
"10.0.0.1 DNS reply size limit is at least 3843"
"Tested at 2010-04-21 11:57:32 UTC"
"10.0.0.1 sent EDNS buffer size 4096"

I guess the alternative would have been to set a more restrictive edns-udp-size as an option in my bind config... eg:

options {
 edns-udp-size 512;
}

But given the way EDNS likes to sometimes have larger packet sizes this just seems inefficient.

I found there was a surprising lack of clear and concise information on this problem online, and given the default message-length when inspecting DNS with a Cisco asa is 512 bytes i'd be interested to hear if anybody else as run into these problems.

Booting a Cisco router from TFTP when the flash: has died

 Recently i had a 3825 router which failed to come back up after a power-down, reason was that the Compact Flash card had died. :(

This left us in a bit of a pickle and i needed to get this backup ASAP.. Solution in this case was to get the router booted from TFTP, lucky in this case i had a serial console on the device.

So at ROMMON i did this:

rommon> IP_ADDRESS=10.1.0.10
rommon> IP_SUBNET_MASK=255.255.255.0

rommon> DEFAULT_GATEWAY=10.1.0.1
rommon> TFTP_SERVER=10.0.0.12
rommon> TFTP_FILE=/tftpboot/c3825-adventerprisek9-mz.124-24.T2.bin

rommon> tftpdnld

! image loads and router comes up with a config which was fortunately save to nvram: which was not affected.