Cisco DHCP issue

[regnez]

Interesting (i.e. strange) issue here...

I have a UC560 serving as both a router for phones and data at an office. Two VLANs setup, 1 for data and 100 for phones. The DHCP scope of the two is

Data: 10.10.29.0 /24
Phones: 10.200.29.0 /24

and things were working fine for awhile. However, recently the UC started to hoard 100+ of the IP addresses within the data scope for itself. Starting at about 10.10.29.20 and going to 10.10.29.150 (ish), it will not hand out those addresses because it itself is using them. From a machine on the network, I can ping any of those addresses and I get back a response. An arp -a shows the same MAC address (the MAC addy of the router) for every IP address in that range.

The problem is that DHCP scope is excluding 10.10.29.1 - 10.10.29.99, because that is where we set static IPs for printers/switches/etc. I cannot for the life of me figure out how to get the router to give up these IP addresses and start handing them out to machines.

Looking at the database (and running-config) on the router, it does not show those addresses as having been handed out nor does it show them assigned to itself.

Thoughts...?

 

[jlazzaro]

IOS version, scrubbed config, and the output of 'show ip dhcp bind' and 'show ip dhcp server stat'

 

[sactwnguy]

What type of cisco router are you using? Is it a Catalyst switch doing the dhcp server? I have seen a similar issue in the past but it had to do with the database not being configured. Can you show your config?

 

[regnez]

What type of cisco router are you using? Is it a Catalyst switch doing the dhcp server? I have seen a similar issue in the past but it had to do with the database not being configured. Can you show your config?

It's a UC560 and it itself is handling DHCP. Configs to follow.

sh ip dhcp bind:

UC_560_Bridgeport#sh ip dhcp bind
Bindings from all pools not associated with VRF:
IP address Client-ID/ Lease expiration Type
Hardware address/
User name
10.10.29.130 0100.24e8.fa85.cc Sep 17 2010 08:35 AM Automatic
10.10.29.232 0100.26b9.eb2a.e7 Sep 17 2010 08:58 AM Automatic
10.10.29.242 0100.26b9.d58f.42 Sep 17 2010 08:49 AM Automatic
10.10.29.248 01c4.4619.6f5d.b2 Sep 17 2010 08:57 AM Automatic
10.10.29.249 01c4.4619.3fa1.33 Sep 17 2010 08:57 AM Automatic
10.10.29.250 015c.5948.ed9b.71 Sep 17 2010 08:57 AM Automatic
10.200.29.100 0158.bc27.747b.33 Sep 16 2010 10:32 PM Automatic
10.200.29.102 01a8.b1d4.fb6d.cf Sep 17 2010 06:54 AM Automatic
10.200.29.104 0158.bc27.751f.98 Sep 16 2010 10:32 PM Automatic
10.200.29.108 0158.bc27.74fb.a2 Sep 16 2010 10:51 PM Automatic
10.200.29.114 0158.bc27.747a.9d Sep 17 2010 07:22 AM Automatic
10.200.29.120 0158.bc27.74fb.c7 Sep 17 2010 07:22 AM Automatic
10.200.29.122 0158.bc27.7520.20 Sep 17 2010 08:07 AM Automatic
10.200.29.124 01dc.7b94.f8ad.6d Sep 17 2010 08:26 AM Automatic
10.200.29.126 0158.bc27.7476.50 Sep 17 2010 08:41 AM Automatic
10.200.29.128 0158.bc27.74fb.34 Sep 17 2010 02:06 AM Automatic
10.200.29.130 0158.bc27.74fb.9d Sep 17 2010 08:07 AM Automatic
10.200.29.132 0158.bc27.7537.45 Sep 17 2010 06:14 AM Automatic
10.200.29.134 0158.bc27.7533.c8 Sep 17 2010 07:25 AM Automatic
10.200.29.136 01dc.7b94.f86e.25 Sep 17 2010 08:29 AM Automatic
10.200.29.137 0158.bc27.74f5.c2 Sep 17 2010 08:11 AM Automatic
10.200.29.138 0158.bc27.7523.79 Sep 17 2010 07:25 AM Automatic

sh ip dhcp stat:

UC_560_Bridgeport#sh ip dhcp server stat
Memory usage 980884
Address pools 2
Database agents 0
Automatic bindings 22
Manual bindings 0
Expired bindings 28
Malformed messages 0
Secure arp entries 0

Message Received
BOOTREQUEST 3
DHCPDISCOVER 153
DHCPREQUEST 4277
DHCPDECLINE 9
DHCPRELEASE 41
DHCPINFORM 575

Message Sent
BOOTREPLY 3
DHCPOFFER 134
DHCPACK 4379
DHCPNAK 21

sh version (just the top):

Cisco IOS Software, UC500 Software (UC500-ADVIPSERVICESK9-M), Version 15.0(1)XA3a, SBTG Special
Small Business Support: http://www.cisco.com/go/smallbizhelp
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Tue 08-Jun-10 13:40 by SBTG

ROM: System Bootstrap, Version 15.0(1r)XA, RELEASE SOFTWARE (fc1)

UC_560_Bridgeport uptime is 1 week, 5 days, 14 hours, 9 minutes
System returned to ROM by power-on
System restarted at 18:50:37 CDT Fri Sep 3 2010
System image file is "flash:uc500-advipservicesk9-mz.150-1.XA3a"

And last but not least, (a heavily modified) sh run:


version 15.0
parser config cache interface
no service pad
service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
no service password-encryption
service internal
service compress-config
service sequence-numbers
!
hostname UC_560_Bridgeport
!
boot-start-marker
boot-end-marker
!
card type t1 0 2
no logging queue-limit
no logging buffered
no logging rate-limit
no logging console
no logging monitor
enable secret 5 $1$PknU$BQRbxk.BC5FerFEsxPdH8.
!
aaa new-model
!
!
!
!
!
!
!
aaa session-id common
!
!
!
clock timezone CST -6
clock summer-time CDT recurring
network-clock-participate wic 2
network-clock-select 1 T1 0/2/0
!
ip dhcp binding cleanup interval 10
ip dhcp excluded-address 10.200.29.1 10.200.29.99
ip dhcp excluded-address 10.200.29.241 10.200.29.255
ip dhcp excluded-address 10.10.29.1 10.10.29.99
!
ip dhcp pool phone
network 10.200.29.0 255.255.255.0
default-router 10.200.29.2
option 150 ip 10.200.29.2
!
ip dhcp pool data
import all
network 10.10.29.0 255.255.255.0
default-router 10.10.29.1
domain-name SES.Local
dns-server 172.16.0.21 172.16.0.22
!
!
ip inspect name SDM_LOW cuseeme
ip inspect name SDM_LOW dns
ip inspect name SDM_LOW ftp
ip inspect name SDM_LOW h323
ip inspect name SDM_LOW https
ip inspect name SDM_LOW icmp
ip inspect name SDM_LOW imap
ip inspect name SDM_LOW pop3
ip inspect name SDM_LOW netshow
ip inspect name SDM_LOW rcmd
ip inspect name SDM_LOW realaudio
ip inspect name SDM_LOW rtsp
ip inspect name SDM_LOW esmtp
ip inspect name SDM_LOW sqlnet
ip inspect name SDM_LOW streamworks
ip inspect name SDM_LOW tftp
ip inspect name SDM_LOW tcp router-traffic
ip inspect name SDM_LOW udp router-traffic
ip inspect name SDM_LOW vdolive
no ipv6 cef
!
!
stcapp ccm-group 1
stcapp
!
stcapp supplementary-services
port 0/0/0
fallback-dn 301
port 0/0/1
fallback-dn 302
port 0/0/2
fallback-dn 303
port 0/0/3
fallback-dn 304
!
!
multilink bundle-name authenticated
isdn switch-type primary-ni
!
!
trunk group ALL_FXO
max-retry 5
voice-class cause-code 1
hunt-scheme longest-idle
!
!
trunk group ALL_T1E1
hunt-scheme longest-idle
translation-profile outgoing PROFILE_ALL_T1E1
!
!
voice call send-alert
voice rtp send-recv
!
voice service voip
allow-connections h323 to h323
allow-connections h323 to sip
allow-connections sip to h323
allow-connections sip to sip
supplementary-service h450.12
sip
no update-callerid
!
voice class codec 1
codec preference 1 g711ulaw
codec preference 2 g729r8
!
!
voice class cause-code 1
no-circuit
!
voice register global


interface GigabitEthernet0/0
description $FW_OUTSIDE$
no ip address
ip access-group 104 in
ip inspect SDM_LOW out
ip virtual-reassembly
load-interval 30
duplex auto
speed auto
!
!
interface Integrated-Service-Engine0/0
description Interface used to manage integrated application modulecue is initialized with default IMAP group
ip unnumbered Vlan90
ip nat inside
ip virtual-reassembly
service-module ip address 10.1.10.1 255.255.255.252
service-module ip default-gateway 10.1.10.2
!
!
interface GigabitEthernet0/1/0
switchport mode trunk
macro description cisco-switch | cisco-switch
!
!
interface GigabitEthernet0/1/1
switchport mode trunk
macro description cisco-switch | cisco-switch
!
!
interface GigabitEthernet0/1/2
macro description cisco-desktop | cisco-desktop
spanning-tree portfast
!
!
interface GigabitEthernet0/1/3
description Interface used to communicate with integrated service module
switchport access vlan 90
service-module ip address 10.1.10.1 255.255.255.252
service-module ip default-gateway 10.1.10.2
!
!
interface Serial0/2/0:23
no ip address
encapsulation hdlc
isdn switch-type primary-ni
isdn incoming-voice voice
isdn supp-service name calling
isdn sending-complete
trunk-group ALL_T1E1
no cdp enable
!
!
interface Vlan1
description $FW_INSIDE$
ip address 10.10.29.2 255.255.255.0
ip access-group 101 in
ip nat inside
ip virtual-reassembly
!
!
interface Vlan90
description $FW_INSIDE$
ip address 10.1.10.2 255.255.255.252
ip access-group 103 in
ip nat inside
ip virtual-reassembly
!
!

interface Vlan100
description $FW_INSIDE$
ip address 10.200.29.2 255.255.255.0
ip access-group 102 in
ip nat inside
ip virtual-reassembly
!
!
ip forward-protocol nd
!
ip http server
ip http authentication local
ip http secure-server
ip http path flash:/gui
ip dns server
ip nat inside source list 1 interface GigabitEthernet0/0 overload
ip route 10.1.10.1 255.255.255.255 Vlan90

I left out all the dial-peers, ephone, and ephone-dn config info (and a bit more), so if you are looking for information that is not there, let me know and I can add it.

:: EDITED FOR LAYOUT IMPROVEMENTS ::

 

[sactwnguy]

I dont see your "ip dhcp database" command. I have seen a similar problem to yours when we didnt have a saved copy of the database after a reboot the switch would think the pool was 100% used. We just point the file to be saved to flash and everything started to work properly.

http://www.cisco.com/en/US/docs/ios/12_0t/12_0t1/feature/guide/Easyip2.html#wp27816

 

[Lithium381]

Message Received
BOOTREQUEST 3
DHCPDISCOVER 153
DHCPREQUEST 4277
DHCPDECLINE 9
DHCPRELEASE 41
DHCPINFORM 575

Message Sent
BOOTREPLY 3
DHCPOFFER 134
DHCPACK 4379
DHCPNAK 21

I've only ever setup DHCP at home in a controlled lab setup....are those kind of numbers normally that lopsided with the REQ / ACKs? Do you get any interesting output from a trace-route? Possibility of proxy-arp going on? Whereby the router serves other hosts using its own MAC address??

 

[Emulex]

can't you just set the phones static and let the FSMO Active directory server handle dns leases (servers run static, everything else on a lease reservation).

do the phones have a DHCP token option? This means without the token they will not take the dhcp from another source. Everyone needs to participate with this method (all phone gear) for it to work.

this is one of the reasons i think the UC series is odd. i want my own firewall, router, dhcp/dns(active directory 2003) - not all their junk.

OER is poor at best with their ISR G2/UC5x0 with static links (small business yo).

I just did cabling and ran separate segments phy so when a machine does a bare metal dump from its raid-0 ssd's (think a few years in advance always when cabling) - it won't affect the phone. Most phones cannot switch 500megabit very well.

btw not to be rude but why not soft-phone it? even thin clients can do soft-phone/PCOIP/etc. seems like a waste to spend $200 on a phone when a whole boxen cost that much to make

 

[regnez]

I've only ever setup DHCP at home in a controlled lab setup....are those kind of numbers normally that lopsided with the REQ / ACKs? Do you get any interesting output from a trace-route? Possibility of proxy-arp going on? Whereby the router serves other hosts using its own MAC address??

Those kind of numbers are not completely normal, but they are not horrifically far off. I did run a traceroute before it was fixed (more on that below) and it didn't show anything fishy.

 

[regnez]

I dont see your "ip dhcp database" command. I have seen a similar problem to yours when we didnt have a saved copy of the database after a reboot the switch would think the pool was 100% used. We just point the file to be saved to flash and everything started to work properly.

http://www.cisco.com/en/US/docs/ios/12_0t/12_0t1/feature/guide/Easyip2.html#wp27816

That command came up blank.

 

[regnez]

can't you just set the phones static and let the FSMO Active directory server handle dns leases (servers run static, everything else on a lease reservation).

do the phones have a DHCP token option? This means without the token they will not take the dhcp from another source. Everyone needs to participate with this method (all phone gear) for it to work.

this is one of the reasons i think the UC series is odd. i want my own firewall, router, dhcp/dns(active directory 2003) - not all their junk.

OER is poor at best with their ISR G2/UC5x0 with static links (small business yo).

I just did cabling and ran separate segments phy so when a machine does a bare metal dump from its raid-0 ssd's (think a few years in advance always when cabling) - it won't affect the phone. Most phones cannot switch 500megabit very well.

btw not to be rude but why not soft-phone it? even thin clients can do soft-phone/PCOIP/etc. seems like a waste to spend $200 on a phone when a whole boxen cost that much to make

The phone DHCP database was not the issue -- they were all working fine. It was the VLAN1 (PCs) database that was problematic.

I agree with you about the UCs, by the way. I hate them to death because of how they handle VLANs vs physical interfaces and a lot of the CLI stuff is bass ackwards. They are (relatively) more affordable than a full-blown setup though, and this is a remote office with maybe 30 people, so anything more pricey would have been hard to justify. I much prefer 2951s and up, though.

And we use physical phones because it'd be near impossible to get a regional VP to agree to softphones to save a few bucks. For 30 (ish) folks, all the phones only run about $6,000, so it's not too bad in the long run. Desk phones head towards obsolescence at a much slower rate than most tech, so $6,000 every 7 or 8 years it not an issue.

 

[regnez]

As an update to this issue, turns out our regional IT guy (I work as a network admin at the corporate office, so I wasn't actually in the office when this all went down) thought it'd be a good idea to go to BestBuy and get a Netgear wireless router for that office.

I am not sure how he set it up or what the exact issue ended up being, but when I got to that office at the end of the day (luckily it's only about an hour and a half away) and saw that, I just about went on a rampage. Replacing the Netgear with a properly configured ProCurve MSM AP, fully clearing the VLAN1 DHCP leases, and power cycling everything involved cleared the issue up.

Many thanks to all for your inquiries and help.

 

[spidey07]

As an update to this issue, turns out our regional IT guy (I work as a network admin at the corporate office, so I wasn't actually in the office when this all went down) thought it'd be a good idea to go to BestBuy and get a Netgear wireless router for that office.

I am not sure how he set it up or what the exact issue ended up being, but when I got to that office at the end of the day (luckily it's only about an hour and a half away) and saw that, I just about went on a rampage. Replacing the Netgear with a properly configured ProCurve MSM AP, fully clearing the VLAN1 DHCP leases, and power cycling everything involved cleared the issue up.

Many thanks to all for your inquiries and help.

Lulz. This is exactly why you don't let anybody attach any network equipment. We've fired people for doing that.

 

[Cooky]

You ever consider BPDU guard, port-security & DHCP snooping so that people can't attach rogue DHCP server or AP?
These are Cisco terms...not sure what HP calls their offering.

 

[VinylxScratches]

Lulz. This is exactly why you don't let anybody attach any network equipment. We've fired people for doing that.

Serious?

It seems once every 3 months, someone plugs a Linksys router on the network with DHCP enabled and it dished out DHCP. It always seems like the last thing to think of until you realize when a user does an IPconfig /all and you see that it's got a 192.xxx.xxx.xxx address lol. Once time we had a user at the regional office just log into the router, kill dhcp, change the password of the router. Left it at that...

 

[regnez]

You ever consider BPDU guard, port-security & DHCP snooping so that people can't attach rogue DHCP server or AP?
These are Cisco terms...not sure what HP calls their offering.

We do use BPDU guard and shutdown any ports that aren't being used, but unfortunately we have a pretty loose reign on our regional IT folks. In this case, the fellow who installed it knew just enough to "get it working," though he's since lost that privilege.

All the switches and routers now have new credentials, so if he tries this again he'd have to change those credentials first, which would be enough to get him fired.

 

[regnez]

Serious?

It seems once every 3 months, someone plugs a Linksys router on the network with DHCP enabled and it dished out DHCP. It always seems like the last thing to think of until you realize when a user does an IPconfig /all and you see that it's got a 192.xxx.xxx.xxx address lol. Once time we had a user at the regional office just log into the router, kill dhcp, change the password of the router. Left it at that...

My workplace is like yours in that someone simply installing a rogue piece of equipment without permission is not enough to get them fired, mostly just a stern "don't do that again."

On the other hand, if we had anyone just log into a router and kill DHCP, HR would have some paperwork to take care of within the hour.

 

[VinylxScratches]

Oh... well this was a home Linksys router, infact it was a wrt54g. Somone engineer put it on the network from what I remember.

 

[alkemyst]

Serious?

It seems once every 3 months, someone plugs a Linksys router on the network with DHCP enabled and it dished out DHCP. It always seems like the last thing to think of until you realize when a user does an IPconfig /all and you see that it's got a 192.xxx.xxx.xxx address lol. Once time we had a user at the regional office just log into the router, kill dhcp, change the password of the router. Left it at that...

You explain this in one of the many things they encounter at hire. If they don't read up on the place they decide to spend the majority of their waking hours at, then they fail at their life.

I can agree at times it makes sense to allow other devices to plug in.

I have found most companies today are trying to do to much to handle "I fucked up at my job today".