[NTLUG:Discuss] Re: OT: Ethernet 5.4.3 rule -- gone, nada, zip ...

[Bryan J. Smith]

On Wed, 2004-08-11 at 18:44, Richard Geoffrion wrote:
> OK...I know this isn't specifically Linux related...but does the old
> Ethernet 5-4-3 rule apply in today's world of ethernet hubs and switches?

No, definitely not, completely different.

- Layer 2 collision domains (concentrator/repeating)

The old 5-4-3 repeat rule for 10Base does not apply to 100Base.
Under 10Base, you can have upto 4 repeats.

Under 100Base, you can only have _1_ repeat!
That means if you add a 2nd hub, that's it.
I.e., you can_not_ do this with 100Base:  

     100Hub1
     /    \
100Hub2  100Hub3

Because it takes more than 1 repeat to go from 100Hub2 to 100Hub3.

Also note that a tranceiver to a different media type is also considered
a "repeat."  So if you have a 100Base-FX or 100Base-LX, etc... in your
100Base hub, then there goes your "repeat."

Stackable 100Hubs use a different mechanism to tie their backplanes
together, so they may get around this rule.

- Layer 2 broadcast domains (bridging/switching)

Now most modern networking equipment is switched.  It doesn't cost
anything as Ethernet ASIC and PHY chips are pennies these days.

10/100 "Dual-Speed" Hubs are a 10Base collision domain and a 100Base
collision domain separated by a Store'n Forward (SnF) bridge.  So it's
like having 10-to-100 bridge with a 10Hub and 100Hub attached, only it
is negotiating 10 and 100 on each port.

Cheap 10/100 switches typically use SnF as well, so there are no "basic"
limitations to how far you can go.

More expensive 10/100 switches may offer cut-through or adaptive
cut-through to reduce the latency over SnF.  It all depends, but if you
have one that offers such, it's best _not_ to

- Early 1000Base-LX/SX adoption and frame drops

At 1500 byte standard Ethernet frames, the following processing in
required:
   10Base:  6600 frames/second
 100Base:  66000 frames/second
1000Base:  660000 frames/second

Most modern 10/100 NICs can handle 66000 frames/second.  In the
mid-'90s, paying for 16-64KB of SRAM (that's 0 wait _static_ RAM, not
SDRAM -- e.g., you'll find SRAM in CPU cache) cost a bit extra and those
were "server" cards.  But today, most NIC ASICs come with 16KB of SRAM
or so so they can handle those rates with few drops.

When 1000Base was first adopted, most people paid for it, typically with
optical cabling.  So then there was the added cost of the LED
(1000Base-SX multimode fiber) or the longer run Laser (1000Base-LX
singlemode fiber) transmitter/receivers.

So the $300-1K+ NICs came with anywhere from 512KB to 2MB of SRAM.  That
typically allowed 1000Base to "get away with" processing 664000
frames/second.  And performance was typically 2-5x 100Base, with minimal
frame drops.

The clear "recommendation" to reach more like 5-8x 100Base was to switch
to 9000 byte jumbo frames, so the number of frames/second went down to
around 110,000 frames/second.

But the kicker in doing that is that _all_ nodes must speak that frame
rate, or the switch must be high-end enough to translate.  Typically
IEEE 802.1q VLANs were used to logically segment 1500 byte and 9000 byte
networks -- but that's a whole other story.

- Commodity 1000Base-T adoption, Frame Drops and EMI issues

Now we have the the proliferation of cheap 1000Base-T equipment and
NICs.  Since the NIC ASICs are really no different than 10/100 ones
before them for cost considerations, they rarely_ put enough SRAM in the
ASICs/switches these days.  Maybe 64KB if you're lucky.

That's why, just like 10 years ago for 10/100Base-T, you see
10/100/1000Base-T "desktop" and "server" NICs.  With the "desktop"
1000Base NICs, "frame drops" are possible, because of the small SRAM
on-board.

Since most people don't want to implement 9000byte jumbo frames, this is
a _real_ issue.

The new "workaround" is to make sure you get equipment and NICs that
support 802.3x (_not_ to be confused with 802.1x) which lets nodes
communicate back to the switch, "whoa, I'm not ready for more!"

Unfortunately a _lot_ of "cheap" switches out there do _not_ support
802.3x.  So be wary!

Then there are the EMI issues.  I haven't seen the final EIA-TIA specs
(I need to read up), but the draft I saw 2 year ago said no more than 48
pairs (12 1000Base-T, since it uses 4 pairs, 8 wires, per run) of
Cat-5e/6 within 5' of each other.

On a home or SOHO network, that's not an issue.  But in a wiring closet,
it's detrimental!  Consider fiber in your closet, or at least for your
servers.  EMI can kill your performance.

- Other Layer 2 issues

There are also other Layer 2 considerations.

802.1d Spanning Tree packets can typically only go through upto 7
"bridges" before they reach their TTL (time to live).    Even if you
aren't using 802.1d, it's typically _on_ by default in many switches.

Putting more bridges (e.g., switches) on your network, especially
farther down in the "leaf" of the network (e.g., inside of people's
cubes), can cause packets to _never_ reach one end of a layer 2
(Ethernet) network from another.  And that's when storms occur, bringing
a network down to its knees.

This is also why you should always _outlaw_ people putting their own
Ethernet switches in their cubes.  If you think this is anal, this issue
hits even "small-to-medium businesses" (SMBs) in the butt all-the-time.

It's very important to segment Layer 3 Broadcast Domains as much as
Layer 2 Collision Domains.  I.e., _Route_inside_ your larger network. 
And when I say "route," I don't mean use a cheap NAT device.  Those can
actually cause "ARP storms" when used internally (NAT is not
firewalling, long story).

In the late '90s, Cisco used to call this its 3-layer
"Core-Distribution-Access" model.  Their newer AAVID model is slightly
different, but it has the same theme -- route _inside_ a network with
more than 100-250 nodes.

> Other than total unmanagability and security issues...what happens to a
> network when little workgroup hubs and switches sprout up all over the
> network so that people can plug in network printers, laptops and who knows
> WHAT into their one office ethernet jack?

The big one is 802.1d Spanning Tree TTL as I detailed above.
There are countless other issues as well.

Layer 2 (Ethernet) has _never_ been designed to be extended beyond a
couple of layers of backbone (note that a "backplane" or "stack" counts
as 1 backbone).  You _definitely_ want to implement layer 3 routing
internally when you pass 100-250 nodes.

Cisco's aged Core-Distribution-Access model is still a "good first
start."  Many other vendors have similar approaches.

-- Bryan J. Smith, CCDP (not that it means anything)


-- 
Time to switch: http://www.mozilla.org/products/firefox/switch.html
-------------------------------------------------------------------
Bryan J. Smith                                   b.j.smith at ieee.org