[NTLUG:Discuss] Re: why swap? -- welcome to Gigabit Ethernet

[Bryan J. Smith]

On Fri, 2004-09-10 at 13:09, Robert Citek wrote:
> Any suggestions on how to test this?

They sell hardware to do such, but it is very expensive.  A better route
would be to use an intelligent switch so it would provide RMON
statistics.  But then the switch would probably be capable of handling
it on its own versus a cheap GbE switch.

The RX-errors listed in netstat would be a good first start.
I myself need to get familiar with the features on the /proc filesystem
that can give you more detailed layer 2-4 information.
I've used intelligent switch fabrics as my crutch for far too long.

You might also try switching to 9000 byte jumbo frames and see if
performance improves.  You'll need to setup a dedicated subnet.  You
could do this with a VLAN if your switch supports it, although be wary
of your internal layer 3 routing infrastructure (if you have any at
all).  In all cases, make sure your switch supports larger than 1500
byte Ethernet frames (nearly fully 802.1Q compliant switches do).

> I imagine connect the two machines directly via crossover cable, push
> a lot of data down the pipe, and measure the rate.  Then repeat via
> the switch.  Or are there better ways?

Loss of performance is just a side-effect of any problem.  They key is
to find out what the _exact_ problem is.

The _least_ideal_ case is when the receiving station cannot handle the
incoming 

> On one machine (10.4.0.5) using the e1000 driver:
> # lspci -s 00:0b.0 -v
> 00:0b.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet 
> Controller (Copper) (rev 01)
>          Subsystem: Intel Corp. PRO/1000 MT Server Adapter

Good, Intel.  They are great with the specifications.

The i82545EM RISC-ASIC only has 64KB of integrated SRAM cache.  That's
fine for a 10/100 "server" adapter, but _not_ 1000Base -- especially
with 1500 byte jumbo frames.

I'm looking up the PRO/1000 MT Server Adapter specifications and it
doesn't look like they add any SRAM to it.  I can't seem to find a model
that does.

So, if you look at the board physically, are there any additional SRAM
chips? 

They will be significant is size.  Just the typical 128Kb x 16 (yes, KB,
not MB) SRAM IC is easily the size of today's 128Mb x 4 SDRAM IC you'd
see on a DIMM.  Like a SDRAM IC, SRAM ICs have a "-X" timing at the end,
but that's the actual read latency too (and not just write sync like
SDRAM).

> On the other (10.4.0.4) using the tg3 driver:
> # lspci -s 01:06.0 -v
> 01:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 
> Gigabit Ethernet (rev 14)
>          Subsystem: Dell Computer Corporation Broadcom BCM5700

Sounds like mainboard integrated.

The BCM5700 has 96KB of SRAM internally, with support for up to 16MB
externally.

> Don't know what the switch is (don't have physical access to it right 
> now).  Will find that out later.

That's the biggy.  Even if the NICs suck, if the switch is good, it can
buffer the traffic.  At least with most setups -- it all depends.

But even if a NIC supports 802.3x, if the GbE switch doesn't (and a lot
of "cheap" ones don't), it doesn't matter, it won't work -- or at least
not ideally (the switch becomes more of a "dumb hub" -- although that's
an oversimplification).


-- 
     Linux Enthusiasts call me anti-Linux.
   Windows Enthusisats call me anti-Microsoft.
 They both must be correct because I have over a
decade of experience with both in mission critical
environments, resulting in a bigotry dedicated to
 mitigating risk and focusing on technologies ...
           not products or vendors
--------------------------------------------------
Bryan J. Smith, E.I.         b.j.smith at ieee.org