Splitting WANs, frame and VPNs

[spidey07]

So here's an interesting idea and I'd love some feedback.

With an ever growing need for bandwidth for a decent sized enterprise with 70 remotes sites all using frame-relay cost and performance are quickly becoming overwhelming. If you want 768-1544 Mbs of frame-relay it is going to cost you. Today our remote sites have anywhere between 256 and 1544 Mbs connections. And even then it is sometimes not enough with a huge bulk of e-mail, spreadsheets and database replication occuring.

What's to prevent me from running some kind of multimegabit DSL service to these sites to ease the load? It complicated, its ugly, but it could offer a heck of a lot of performance.

Pros with Internet based VPN - low cost, high bandwidth
Cons with Internet based VPNs - public network, no QoS, reliability, routing complexity

Pros with Frame - private network, congestion management, QoS, cheaper than private line
Cons - too expensive for multimegabit, cost

I figured I could route all bulk transfers like e-mail, Database replication and what not over the VPNs (I alreay have the hardware and infrastructure to do this), so these two networks Frame and VPNs would be application aware. Policy routing would handle any decisions I need it to. The cricital/interactive apps that need to work in real-time would be sent on the frame with all the goodie, goodie QoS and latency guarantees.

But then comes the part about integration and failover and it gets messy real fast. Not to mention security issues.

Anybody have any ideas or have you heard about large companies replacing or augmenting their WANs with VPNs? Or should I be inquiring about a MPLS backed VPN?

thanks for reading.

Guess I need to explain a little more. Each site would have two connections - a DSL/VPN and a frame-relay all wrapped up in a single router to make routing easier.

 

[Garion]

I've done it, mostly with the VPN as a backup to the frame. Not too difficult, just a floating static out the ethernet interfaces. I did not, however, try to do anything like L4 routing.. Sounds very, very messy.

A few things to keep in mind:

Availability of DSL is pretty limited - We hit about 60% across the country, and a lot of it was ADSL, where uploads were limited to ~512Kb, even when we could get a massive download speed.

It might seem cheap, but you need to factor in the cost of your hub site Internet circuits, the redundancy required there, firewall upgrades, etc. This costs more than most people might think and it's possible your VPN traffic could affect your general user browsing experience if it gets too busy.

You also have to consider the cost of the tunnel servers on both ends - For 60 sites, you're looking at a good chunk of change.


The only way that I can see you pulling this off would be to use some kind of packet shaper at the home office to deliver xx% of the traffic to the remote office through the VPN and let the rest of the traffic go across the frame. Let the remote office router use the frame as their default route, so they don't eat up the small upload cap of the DSL connection. God help you if someone else has to troubleshoot it, however.

- G

 

[spidey07]

It might seem cheap, but you need to factor in the cost of your hub site Internet circuits, the redundancy required there, firewall upgrades, etc.

This is actually already covered. We have two 3030 concentrators and two DS3s plus the security structure to do it.

The complexity is what is bothering me. I like things simple so the NOC and support staff can support it.

 

[ScottMac]

For spans of 100 miles or less, you may be able to get gigabit service from your telco (in the Midwest, SBC/Ameritech calls it "GigaMAN." Basically two JDS Uniphase boxes with some fiber in the middle. It's actually an excellent deal price-wise. IN some cases, I've heard it's cheaper than a DS3 (and absolutely cheaper from a hardware perspective: JDSU boxes cost less than many of the blades for a DS3 MUX or router).

You may be able to negotiate for dark fiber as well.

You could register as a CLEC and purchase wholesale bandwidth up to and including OC192s (Cheap, Honest!) The downside there is that if someone wanted to buy bandwidth from you, you'd have to sell it to 'em.

Private Microwave is rearing it's ugly head again. A single span can support up to (at least) three DS3s. For a MAN setup, there are a couple companies that offer "cellular" microwave for path redundancy.

VSAT or USAT might also work for you as well. The bandwidth can be scheduled such that your daytime bandwidth is ~64/128 but your nighttime bandwidth is multimegabit (check out Hughs Corp, they do this very well, and they're dealing Mofo's when it comes to transponder time).

Gotta run - just tossing in my .02

Happy New Year! (Did you guys get your shirts?)

Scott

 

[Buddha Bart]

Have you priced it out much? I took a peek over at dslreports.com for business class sdsl lines. Seems that 1 - 1.5mbps lines go for about $350, but I'm sure it would vary widely by area.

But then comes the part about integration and failover and it gets messy real fast.
Couldn't that be handled by whatever routing protocol your using? I've only worked with rip & igrp (i'm still just shy of ccna), but I know at least with IGRP it would detect the failure and expire the route across your VPN, leaving only the route across your frame relay for it to chose from. Or was it something like 90 for an update, and three failures to update it'll expire the route... so maybe a full 5 minutes of downtime. I forget the exact numbers, but I do remember you can change them yourself, so you could set those routers to update each other more frequently. Frankly all this is from my vauge recollections of IGRP and I think I remember hearing everyone uses ospf anyway.

Not to mention security issues.
Whatcha mean? I thought IPsec was considered "enough". Just out of curiosity, would your routers be able to handle the extra load of all the encryption for the VPNs?

bart

 

[cmetz]

Another con to be aware of is MTU. It's a lot better for performance if all links in your paths have the same MTU, and thus IP fragmentation and reassembly simply never happen. (yes, modern systems are supposed to have Path MTU Discovery.... but not all do, it's not quite as simple in practice as in theory, and non-TCP apps mostly don't handle it well)

Do not for a moment underestimate the con of the added complexity burdening your support staff. If your folks can't keep it working reliably, you aren't saving the company real money (yes, YOUR budget has a savings, but the lost productivity is a serious, though sadly hidden, cost to the business). This is my biggest gripe with the whole IT field, but it's a whole different discussion for a different forum.

All that said....

Frame does seem to be seriously on the way out. IP/Internet access solutions like DSL are very competitive and that competition has brought prices way down, and traditional telco WAN solutions have not come down to match. It's just hard to compare frame against SDSL, or especially ADSL, and not think you're paying a whole lot more money for a whole lot less speed. And the gap is likely to widen in the long run. More and more companies are dumping frame for IP VPN services.

There are basically two major approaches you could take here:
1. Buy public Internet circuits (DSL) and then use IPsec VPN devices you run
2. Buy a managed service

There are a lot of different managed "VPN" services out there, and what they do and give you varies wildly. Some folks are just doing IPsec VPNs, some are doing ATM/Frame/MPLS VPNs, some are doing separate circuits, and some... who knows. The downside to all these managed services is cost. The upside to all these managed services is that, if you get one that's well regarded, you can probably decrease the complexity load on your own support people and make it Somebody Else's Problem.

I personally would not recommend you run both frame and DSL and then try to use the two in any sort of load balancing/inverse multiplexing configuration. It is possible, but unless you have a networking wizard on staff, you're just signing up for a whole mess of headache. You can use it for redundancy/fail-over more easily, but then you should consider whether your goal is reliability or cost (the two are usually at odds with each other). If you do try to do something like this, use OSPF. RIP and IGRP are evil. OSPF is pretty straightforward to set up and doesn't have the protocol-level flaws/gotchas.

If you do DSL, you will likely only be able to migrate some portion of your sites to the new setup, because there's no way you'll have 100% availability. It totally depends on where you are as to what the coverage will be like. I would suggest you go pre-qualify a bunch of sites to get a rough idea of how many you could move to DSL. If you could move 75% of your sites to DSL, it's probably a great idea, but if you can only move 25% of them, then it's probably not.

The other piece of advice I'll give you is that I would strongly suggest you find an ISP who connects to multiple DSL CLECs and try to use them exclusively for any IP VPN service. Crossing ISPs almost always is a recipe for added latency and packet loss. Staying within one ISP's network is a lot happier place to be. Having CLEC diversity is useful because inevitably there will be some COs that only some of the CLECs are in, plus you have more choices in terms of SDSL and ADSL speeds.

Trying to set up as a CLEC yourself, an ISP yourself, or to use dark fibre or microwave are all interesting ideas but again unless you have at least one real networking wizard you're probably going to find it's more headache than anything else given the 70 site scale. At that scale, you need to have a small number of cookie-cutter configurations that you can deploy at most sites.

 

[spidey07]

Thanks a bunch for the feedback. Fortunately (or unfortunately) I am the Director of Network Systems. Meaning my guys do it all transport and communications wise.

Unfortunately I'm also the guy that writes the budget and gets bonuses for doing more with less, directly tied to my bottom line. I can't stop the application folks from writing god awful crappy code (i've tried for 5 years) so I have to live with VPs griping about network response time and speed. "Well no bull that 2.5 GB database is taking so long to replicate and killing the atlanta office. what do you want me to do? pull bandwidth out of my heinny?"

yes, YOUR budget has a savings

Hence why I like this idea.

BUT - technically speaking think about running some sort of application aware, policy based routing with private frame-relay and IP/VPN services. I don't even want to think about it. Trying to maintain route states and reachibility over an IPsec tunnel makes my skin crawl. I'd probably be the only one who could support it and that is not in my job description.

ISP would probably be AT&T and AT&T only. I like the idea of keeping on one carriers backbone.

Deep down I think this is a bad idea, but given the current conditions of a FAT WAN (DS3 and T1s everywhere) getting taxed I have to do something. If only I could get other departments to fix their applications and realize that you are NOT SUPPOSED TO MOVE 100s OF MEGS OVER A WAN.

Bad developer, go back to you hole.

-edit- everything here is long-haul, over 100 miles.

 

[Garion]

By the way.. Do most of your offices happen to use the same telco? If so, can you negotiate a private DSL network from them? We used to use GTE/Verizon and they were able to bring in a bunch of DSL lines and aggregate them onto a DS3 for us. We used it for home user connectivity, but it would work the same way for your offices. I believe there's a couple of other providers that can do this nationwide - Might check with Covad and see what they can offer.

No use using the Internet if you can stick with one carrier's private network and let them handle the transport, too. Much cheaper in the long run.

- G

 

[Goosemaster]

It seems that security hasn't been covered at length. Obviously iff it is not a problem, DSL is fine.

Even so, VPNs aren't as safe as Frame Relay. Since only the telco's can look at it before it goes in the "cloud" you minimize interception.

Also, have you considered PP options that aren't Frame, such as t1, t3 and the DS varieties?

 

[spidey07]

I'm looking at all options on increasing WAN performance. Problem is trying to price it out on the large scale to look at the final numbers. I need to get the options down and price the total project and recurring costs for 5 years, including 18 month reviews, etc. YUCK!

Point-to-point leased line is attractive but expensive. But then again prices have come down.
Private DSL is attractive
IP based VPNs
All Frame
Frame/PTP mixed
Frame/VPN mixed

satellite = latency too long

In many sites a T1 isn't enough. So its really "provide top notch performance for no expense".

 

[Goosemaster]

Originally posted by: spidey07
I'm looking at all options on increasing WAN performance. Problem is trying to price it out on the large scale to look at the final numbers. I need to get the options down and price the total project and recurring costs for 5 years, including 18 month reviews, etc. YUCK!

Point-to-point leased line is attractive but expensive. But then again prices have come down.
Private DSL is attractive
IP based VPNs
All Frame
Frame/PTP mixed
Frame/VPN mixed

satellite = latency too long

In many sites a T1 isn't enough. So its really "provide top notch performance for no expense".

no offense man, but you're screwed.


Seems like VPn'ing through DSLgives you the most bandwidth. Maybe there are some QoS options you can somehow get? wasn't aware of any for DSL but maybe you'll get lucky.

 

[cmetz]

A friend of mine has a small ISP/ASP and I help them out with advanced stuff from time to time. We examined in depth the idea of providing DSL service, and our totally unsurprising conclusion is that it's really not a good place to be unless you've got a huge scale. If, say, 35 of your 70 sites could get DSL, then it probably still would not make sense for you to try to pull DSL L2 into your data center and be the "VPN ISP" yourself, it would be more cost-effective to buy a full-blown DSL L3 package from some ISP and L3 VPN them together. It would have costed us more to get a DSL L2 circuit provisioned, not counting our interconnect line to the CLEC, than it costs us to buy a full blown DSL L3 Internet connection. Then add our interconnect, our equipment costs, and the costs of setting up the provisioning infastructure. It was a bad deal, so we don't do that. (also, if you get public L3 Internet connections, you can then build cut-through tunnels and not make random sites as dependent on your central site)

AT&T bought out Northpoint's assets, but it's really not clear what they did with them. AT&T does offer DSL service. I do know that several area COs that were up Northpoint are not available for AT&T DSL. But anyway, AT&T's IP network is a good one, and it would for example be very reasonable to get DSL circuits where available and IP over T1 and/or Frame circuits where not and to do an IPsec VPN between everywhere. AT&T should have sales critters eager to help you with this.

Now, QoS and DSL... forget it. DSL in practice is not considered a reliable service. Now, in practice, good business-class DSL from a reputable provider should be okay, and it's great for the cost, but make no mistake that you will get no useful COMMITTMENTS (ones you can make stick) out of anyone about DSL's reliability/performance. You get what you pay for. If you need five-nines uptime and written committments, buy point to point T1s. Even they have maintenance, though, and no the telcos don't tell you in advance, they just don't count that towards their SLA downtime.

You might be able to use an approach like getting a high-speed DSL line and then setting up an on-demand 128k ISDN lifeline. AT&T offers ISDN data call long distance, it's very expensive, but if it's occasional use only and better than downtime... well then. Pull BRIs to remote sites and a PRI into your main site, and you can set up an ISDN "dialup" backup network. You could do exactly the same with modems, potentially even bonding them together, though I wouldn't want to be the one selling that to management!

The main wins with frame are that it's a closed network (your average bored teenager can't get in, unlike the public Internet), and that the telco folks usually put more design behind the CIRs. The main loses with frame are that it is still packet-switched and suffers all the issues of a packet-switched net (yes, there can be oversubscription especially in terms of IIRs, and large telcos' frame nets have melted down before), and cost. Again, my assessment is that Frame is going away. I would only recommend using it as a way to get from an end site to an ISP's POP when you don't have any more cost effective option. (Frame is a cost effective way to make a "T1" hop over a few COs, for example)

The other approach you might consider is to go look at ATM. I hate ATM. ATM is evil, horrible stuff. But you see, a lot of telcos built big elaborate ATM networks when it was The Next Big Thing, and there aren't many ATM customers. So some telcos were offerring ATM service at fire-sale prices. Then DSL came along, based on ATM, and now I don't know where things are. In particular ATM might be interesting when you need multi-megabit burst speeds. It probably would not be interesting at less than T1 speed.

For security, if it really matters to you, always do full 3DES+HMAC SHA1 IPsec from site to site regardless of the underlying transport. That's as good as you're going to be able to get with normal commercial grade stuff. (If you have the kind of security requirements that seriously exceeded that solution, you really shouldn't be asking what to do here!)

For reliability, the IP world is kinda tough especially when you're on a budget... and isn't everyone? The best approach I've found is to have a good backup circuit that goes somewhere substantially different. We have several customers set up with a point-to-point T1 and a DSL back-up, using some special routing magic we developed (basically nobody doing DSL will do BGP over it), and that works pretty well. Getting, say, two T1s to the same site turns out to not be as reliable, because when the telcos are messing with one, they often are messing with the other the same way. So if you try to be redundant, being as different as possible is good.

The other thing you might seriously do is see if you can loan some of your network engineering folks to your app folks and try to help them find and implement ways to decrease their load on the network by changing the way they do things. For example, rsync (rsync.samba.org) is a wonderful tool. I understand that most org charts have IT being a totally different group than everyone else, but working together closed-loop can make things better for everyone.

 

[Garion]

I hate to say this, but you might consider taking a different approach to this problem. There's an old saying that fits this situation perfectly - "If you build it, they will come". Microsoft is a wonderful example. The faster our PC's get, the bigger, more complex, and slower their software gets.

You probably need to sit down and take a very hard look at the traffic going across your WAN to really figure out what's going on. If you keep throwing bandwidth at it without knowing exactly what's causing it, you are simply feeding the monster, not taming it.

Sometimes application developers simply refuse to believe that they have problems with their code, and unless you can prove it to them and explain it to them in a language they will understand, you'll never get anywhere. Been there, done that, and didn't even get the T-Shirt.

One tool that we've found to be absolutely invaluable to analyze application performance is Compuware's Application Expert.. It takes sniffer technology and breaks the app up into it's component parts and analyses all of the communications that happen - DAtabase queries, etc. and can show an incredible level of detatil. It also provides information on the amount of bandwidth requred for the application, it's sensitivity to latency, etc. Waaaay cool stuff. I don't have it, but we have a whole team of guys that do nothing but use it to analyize apps within the bank and help them optimize them for performance and network efficiency.

One note - It's very expensive, but we've found it to be worth it. Might be worth a gander.

- G

 

[spidey07]

Garion,

[rant on]
I've done countless studies and analysis on exactly what the traffic is. It is all lotus notes database replication. We have 100s of databases ranging from 5 MB to 100 GB. I've had countless meetings with the director of application development. Still when you have power users writing the notes applications/databases there is little you can do in terms of managing the stupidity of somebody who thinks a good idea to write a process database and use 2 MB images in a database that contains 100s of said images on how to turn a screw (not kidding on this one).

These users in no way shape or form report to IT, so there in lies the problem. There is no penality for their actions and network services is on the receiving end of "why is ITs network so slow? This is unacceptible".

upon pointing out the problem application and explaining why it is a bad idea to move 100s of MB across a WAN, IT is told - well your network sucks if it can't run this application. And that comes from my CIOs boss.

hands tied...need to throw bandwidth at it...been fighting it for years.

[rant off]

edit - as an aside our notes server farm has over 6 terabytes of storage, and we're out growing it.

 

[Buddha Bart]

wouldn't it be funny if this was all because none of them could understand .gif?

I imagine if they're techical documentation diagrams they're either not color or very few colors, as well as probably having lots of hard edges. This is the kinda stuff .gif does amazing work on.

Unless everyone involved is in a constant state of needing to print stuff at 2400dpi. Though in that case I bet you'd be bitching more about your print servers

Honestly, if its worth it to you, try throwing out an olive branch. Ask them for like 3 hours of one of their developers time to explain the whole application out to you or one of you guys, and if you think there's a smarter way of implimenting it so it doesnt hurt the network so bad, let them estimate how much work it would take and offer them a budget transfer for the cost. With bandwith being a recurring cost, and having 70 sites, even if you had to pay for them to outsource it to better developers it would probably be worth it. Plus, if you can prove all this with numbers, you might not even have to pay for it, just prove how it'll cost the company less overall to the right higher-up (where your boss and their boss intersect so the person doesnt really have a stake in either side).

bart

 

[SR]

Before commiting to this solution role it out to a few sites to see how it works. More importantly is the extra entry points into your corporate network. Will you be tunneling everything (the apps specified) though the firewall/vpn concentrators or will you allow local network internet traffic flow through the firewall as well? Does Cisco support vpn concentrator to vpn concentrator using AES 128/256? I know they do on client to concentrator connections. I know cisco has or is comming out with cisco work managemnt for pix so that might be helpful or managing the pix without using cspm. However Cisco, in my opinion, has better vpn management when using routers instead of pix or vpn concentrators.

You could employ load balancers (radware, etc) instead of policy routing on the local lan for greater redundancy.

Another thought would be hosting some of your apps on citrix servers to achieve some bandwidth savings?

Another question is are why is their so much replication happening?