Hello all,

At the University of Michigan we just had Robert Colwell (formally of Intel)
come in and give a talk. After the talk one issue came up which I found I
was disagreeing with everyone about (generally a sign I'm wrong of course).

Here is the question:
What is the expected trend in bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a GHz or
will see a steady increase? And if an increase, where do we think the
number will be in 5 years?

Related:
What limits point-to-point bandwidth per pin? It it power, cost, or just
raw physical limits? Something else (transistor switching speed?)

What does the cost per signal pin look like? Put another way, at a
reasonable price, what is the bandwidth trend for point-to-point
communication?

I did a bit of a paper search (~1 hour) and turned up very little. Hot
Interconnects seemed to have a fair bit, but no real trends or predictions I
could find.

Mark

From:	David Wang
Subject:	Re: point-to-point busses
Date:	Mon, 10 Jan 2005 16:40:48 +0000 (UTC)

Mark W Brehob wrote:
> Hello all,

> At the University of Michigan we just had Robert Colwell (formally of Intel)
> come in and give a talk. After the talk one issue came up which I found I
> was disagreeing with everyone about (generally a sign I'm wrong of course).

> Here is the question:
> What is the expected trend in bits/second/wire over a point-to-point
> connection (off-chip)? Do we expect to see a cap of just over a GHz or
> will see a steady increase? And if an increase, where do we think the
> number will be in 5 years?

> Related:
> What limits point-to-point bandwidth per pin? It it power, cost, or just
> raw physical limits? Something else (transistor switching speed?)

> What does the cost per signal pin look like? Put another way, at a
> reasonable price, what is the bandwidth trend for point-to-point
> communication?

> I did a bit of a paper search (~1 hour) and turned up very little. Hot
> Interconnects seemed to have a fair bit, but no real trends or predictions I
> could find.

These trends or projections can be found at ITRS's web site.

You have to dig around to find what you need, but here's an example.

http://www.itrs.net/Common/2004Update/2004_000_ORTC.pdf

In table 3b (page 17 of 29), you can find the number of pads expected
for a microprocessor, the number signal I/O expected for a few years
down the road. Scroll down to the next page, you can find the expected
cost ranges per pin for expensive high frequency processors/ASICs as
well as commodity memory packaging parts.

Keep on scrolling and you can find more projected trends for cost
and frequency. So that should be what you're looking for.

--
davewang202(at)yahoo(dot)com

From:	Paul Rubin
Subject:	Re: point-to-point busses
Date:	10 Jan 2005 02:13:56 -0800

Mark W Brehob writes:
> What is the expected trend in bits/second/wire over a point-to-point
> connection (off-chip)? Do we expect to see a cap of just over a GHz or
> will see a steady increase?

I thought we already had 10 gbits or so. Is that surprising?
We have 600 gbit long haul optical, right?

From:	George William Herbert
Subject:	Re: point-to-point busses
Date:	10 Jan 2005 11:58:13 GMT

Paul Rubin wrote:
>Mark W Brehob writes:
>> What is the expected trend in bits/second/wire over a point-to-point
>> connection (off-chip)? Do we expect to see a cap of just over a GHz or
>> will see a steady increase?
>
>I thought we already had 10 gbits or so. Is that surprising?
>We have 600 gbit long haul optical, right?

Key word there is "optical".

Copper wires are less cooperative.

The current standard "fast" long haul
network is OC-192 at 10 Gbps.

We're in the 40 Gbps range for cutting
edge long-haul networks right now, with
160 Gbps per channel technology in test
but not in common use, and terabit-range
stuff in the labs and a few long distance
demos (using multiple slower channels).

-george william herbert
gherbert@retro.com

From:	Pekka Pietikainen
Subject:	Re: point-to-point busses
Date:	Mon, 10 Jan 2005 18:56:59 +0000 (UTC)

In article <41e26dd5$0$71989$c0de7616@dsl.net>, George William Herbert wrote:
>>I thought we already had 10 gbits or so. Is that surprising?
>>We have 600 gbit long haul optical, right?
>
>Key word there is "optical".
>
>Copper wires are less cooperative.
>
>The current standard "fast" long haul
>network is OC-192 at 10 Gbps.
If I understood things correctly, if you account price the game gets a bit
more interesting. It wasn't too many years ago that the most cost-effective
way of transmitting lots (== more than Ethernet gives you) of bits short
distance (<100m) was copper (and just adding more pairs to get more
bandwidth), and even with fibre say 4*2.5Gbps (possibly using WDM, but even
with separate fibres) being more cost-effective than say 1*10Gbps. Long-haul
is a completely different game, sure, there you can get away with expensive
optics much more easily.

Not sure about the current economics, though :-)

From:	Tim Clacy
Subject:	Re: point-to-point busses
Date:	Mon, 10 Jan 2005 16:40:25 +0100

Mark W Brehob wrote:
> Hello all,
>
> At the University of Michigan we just had Robert Colwell (formally of
> Intel) come in and give a talk. After the talk one issue came up
> which I found I was disagreeing with everyone about (generally a sign
> I'm wrong of course).
>
> Here is the question:
> What is the expected trend in bits/second/wire over a
> point-to-point connection (off-chip)? Do we expect to see a cap
> of just over a GHz or will see a steady increase? And if an
> increase, where do we think the number will be in 5 years?
>
> Related:
> What limits point-to-point bandwidth per pin? It it power, cost,
> or just raw physical limits? Something else (transistor switching
> speed?)
>
> What does the cost per signal pin look like? Put another way, at a
> reasonable price, what is the bandwidth trend for point-to-point
> communication?

Here's an interesting paper on RF interconnect that proposes data rates of
100Gbps per line (and, therfore, up to 20Tbps for 200 lines). The paper
focuses on inter-chip and intra-chip interconnect but should work equally
well over much larger distances.

From:	Tim Clacy
Subject:	Re: point-to-point busses
Date:	Mon, 10 Jan 2005 16:45:58 +0100

....better late than never:

http://www.ee.ucla.edu/faculty/papers/vwani_IEEEproc_apr01.pdf

From:	Mark W Brehob
Subject:	Re: point-to-point busses
Date:	Wed, 12 Jan 2005 01:49:40 GMT

Tim Clacy wrote:

Thanks Tim. Sounds perfect.

I'm due to present a paper soon to a reading group. If I do this one, I'll
post the slides or summary from my presentation.

Mark

> ...better late than never:

> http://www.ee.ucla.edu/faculty/papers/vwani_IEEEproc_apr01.pdf

From:	del cecchi
Subject:	Re: point-to-point busses
Date:	Wed, 12 Jan 2005 22:08:35 -0600

"Mark W Brehob" wrote in message
news:Um%Ed.145$UN1.107@news.itd.umich.edu...
> Tim Clacy wrote:
>
> Thanks Tim. Sounds perfect.
>
> I'm due to present a paper soon to a reading group. If I do this one,
I'll
> post the slides or summary from my presentation.
>
> Mark
>
> > ...better late than never:
>
> > http://www.ee.ucla.edu/faculty/papers/vwani_IEEEproc_apr01.pdf
>
>
I would think you might choose something more related to current and
near future practice. I found it difficult to take that paper
seriously. But that's my old fogey speaking I guess. Actually an
interesting subject and example of wretched excess is the 802.3AN or
10GbaseT effort to transmit 10G ethernet over 50 to 100 meters of CAT6
Unshielded Twisted Pair cable. One could probably teach a semester
course just covering all the topics in that standard (eventually to
be ).

del cecchi

From:	MitchAlsup at aol.com
Subject:	Re: point-to-point busses
Date:	18 Jan 2005 09:11:18 -0800

"What limits point-to-point bandwidth per pin?"

The size/complexity of the pin electronics--given that you engineer the
rest of the signal conduit.

On the output side, consider what it means to assert a signal. In
classical digital logic, the signal must pass certain fixed voltage
thresholds. On the input side, then compare how careful the input
discrimination must be so that the receiver gets the signal that the
transmitter asserted. As these thresholds are made closer to each
other, it is easier to assert a series of signals, but it becomes
correspondingly harder to figure out what the wiggling signals mean at
the receiver. After a while, you transit from digital signaling into
radio signaling where the digital signals are coded onto a carrier
signal with certain properties that makes the receivers job doable. So,
there is a spectrum of signaling techinques that connect the dots (as
it were).

Then, you did not specify what electrical environment these signals
will traverse. There are board issues (FR4 versus teflon; electrical
environmental issues) connector issues, electrical isolation issues,
discontiguous grounding issues. Compaing the data rate achievable by
two chips topologically adjacent and soldered down on a single board is
vastly different than a chip->connector->accross board->connector->coax
cable->connector->different board->connector->other chip. The former is
limited only by chip electronics and the ability to test the devices,
while the later is limited more by connector and electrical issues
rather than pin electronic issues.

If you consider the pin electronics to be a tri-state gate (output) and
an inverter (input) then you are basically limited to the 1.5 GHz limit
for chip-connector-board-connector-chip transmission. And this will
slowly grow.
If you add precompensation to the output electronics, 6 GHz to 10 GHz
will be achieved in the not to distant future.
If you add coding to the output electronics, 10-20 GHz will be
achieved.
If you use a code that looks and smells like a radio carrier frequency,
you can wiggle the pins around 100 GHz data transmission rate
If you add a differential input to the receiver, you can achieve
usefully low error rates up into the 2 GHz range--clock jitter limted
If you add clock recovery to sets of receivers, you can achieve useful
error rates into the 5 GHz range--limited by skew tollerance
If you add clock recovery to each receiver, you can achieve useful
error rates up into the 40 GHz range
if you add extensive signal processing to the input before converting
back to digital, you can achieve useful error rates up into the 100 GHz
range

So, by the time an output pin looks like a complete radio transmitter
and an input pin looks like a complete radio receiver you can indeed
approach 100 GHz of pin to pin bandwidth. However, you will not be able
to get very many of these pins on your part.

Mitch

From:	del cecchi
Subject:	Re: point-to-point busses
Date:	Tue, 18 Jan 2005 18:52:42 -0600

wrote in message
news:1106068278.837190.102020@c13g2000cwb.googlegroups.com...
> "What limits point-to-point bandwidth per pin?"
shannon's law.
>
> The size/complexity of the pin electronics--given that you engineer
the
> rest of the signal conduit.
>
> On the output side, consider what it means to assert a signal. In
> classical digital logic, the signal must pass certain fixed voltage
> thresholds. On the input side, then compare how careful the input
> discrimination must be so that the receiver gets the signal that the
> transmitter asserted. As these thresholds are made closer to each
> other, it is easier to assert a series of signals, but it becomes
> correspondingly harder to figure out what the wiggling signals mean at
> the receiver. After a while, you transit from digital signaling into
> radio signaling where the digital signals are coded onto a carrier
> signal with certain properties that makes the receivers job doable.
So,
> there is a spectrum of signaling techinques that connect the dots (as
> it were).

Fanciest coding in the world can't beat shannon. And you better have
enough high frequency capacity. Remember, the more you modulate the
signal, the wider the bandwidth. Ahh, bessel functions. I remember
them well.
>
> Then, you did not specify what electrical environment these signals
> will traverse. There are board issues (FR4 versus teflon; electrical
> environmental issues) connector issues, electrical isolation issues,
> discontiguous grounding issues. Compaing the data rate achievable by
> two chips topologically adjacent and soldered down on a single board
is
> vastly different than a chip->connector->accross
board->connector->coax
> cable->connector->different board->connector->other chip. The former
is
> limited only by chip electronics and the ability to test the devices,
> while the later is limited more by connector and electrical issues
> rather than pin electronic issues.

The former is still limited by the bandwidth of the channel and the
signal to noise ratio. Going between chips has noise which limits the
number of bits in a baud, and bandwidth limitations caused by parasitics
and device limitations that limit the bandwidth. Testing has nothing to
do with it.

I agree that if the channel is crap, it makes life difficult.
>
> If you consider the pin electronics to be a tri-state gate (output)
and
> an inverter (input) then you are basically limited to the 1.5 GHz
limit
> for chip-connector-board-connector-chip transmission. And this will
> slowly grow.

You talking about single ended NRZ signaling? Where you get the 1.5 GHz
bidness? you mean 1.5 gbit/sec or 3.0 gbit/sec?

> will be achieved in the not to distant future.
> If you add coding to the output electronics, 10-20 GHz will be
> achieved.
10 Gb/s already, no fancy coding. See CEI or UXPI among others.

> If you use a code that looks and smells like a radio carrier
frequency,
> you can wiggle the pins around 100 GHz data transmission rate

And the signal will disappear into nothing before it gets an inch or
two.

> If you add a differential input to the receiver, you can achieve
> usefully low error rates up into the 2 GHz range--clock jitter limted

You were doing all that single ended? Now ISI is a real problem.

> If you add clock recovery to sets of receivers, you can achieve useful
> error rates into the 5 GHz range--limited by skew tollerance
> If you add clock recovery to each receiver, you can achieve useful
> error rates up into the 40 GHz range

Clock recovery or clock forwarded makes no difference. And building
those differential nets to have no mode conversion is a real trick at 40
GHz. Length match to a fraction of a picosecond? Better not turn any
corners.

> if you add extensive signal processing to the input before converting
> back to digital, you can achieve useful error rates up into the 100
GHz
> range

And how are you building a signal processor that handles 100 GHz input
bandwidth?

>
> So, by the time an output pin looks like a complete radio transmitter
> and an input pin looks like a complete radio receiver you can indeed
> approach 100 GHz of pin to pin bandwidth. However, you will not be
able
> to get very many of these pins on your part.
>
> Mitch

I don't believe a word of it. Not with copper interconnect you can't.
and OC 768 which is a mere paltry 40Gb/s takes everything folks know how
to do on a stand alone chip.

del cecchi
>

From:	Thomas Womack
Subject:	Re: point-to-point busses
Date:	19 Jan 2005 20:23:49 +0000 (GMT)

In article <355qq4F4ho84kU1@individual.net>,
del cecchi wrote:

>> If you use a code that looks and smells like a radio carrier
>frequency,
>> you can wiggle the pins around 100 GHz data transmission rate
>
>And the signal will disappear into nothing before it gets an inch or
>two.

What's causing this dissipation? Is it a matter of copper being
"opaque" in the same way it is opaque in the optical, in which case
are there materials which are transparent -- 500THz carrier waves
modulated at 2GHz seem to travel quite happily in quartz?

Is it that copper is dispersive to the point that the upper and lower
side-bands travel at different enough speeds to smear out the signal
entirely?

Or is it some other non-obvious consequence of Maxwell's equations?

As you may have noticed, I am very ignorant of RF engineering; sorry
if these are stupid questions.

Tom

From:	Del Cecchi
Subject:	Re: point-to-point busses
Date:	Wed, 19 Jan 2005 15:16:41 -0600

Thomas Womack wrote:
> In article <355qq4F4ho84kU1@individual.net>,
> del cecchi wrote:
>
>
>>>If you use a code that looks and smells like a radio carrier
>>
>>frequency,
>>
>>>you can wiggle the pins around 100 GHz data transmission rate
>>
>>And the signal will disappear into nothing before it gets an inch or
>>two.
>
>
> What's causing this dissipation? Is it a matter of copper being
> "opaque" in the same way it is opaque in the optical, in which case
> are there materials which are transparent -- 500THz carrier waves
> modulated at 2GHz seem to travel quite happily in quartz?
>
> Is it that copper is dispersive to the point that the upper and lower
> side-bands travel at different enough speeds to smear out the signal
> entirely?
>
> Or is it some other non-obvious consequence of Maxwell's equations?
>
> As you may have noticed, I am very ignorant of RF engineering; sorry
> if these are stupid questions.
>
> Tom

Copper has resistance which causes loss. The resistance increases
approximately as the square root of the frequency due to the "skin
effect". So, yes the copper in a sense becomes opaque. The energy in
the signal gets dissippated in the copper and doesn't get to the load.
We in electric land have have nothing with the low loss, wide band
capabilities of a nice graded index fiber.

del

From:	Terje Mathisen
Subject:	Re: point-to-point busses
Date:	Thu, 20 Jan 2005 08:17:53 +0100

Del Cecchi wrote:
> Copper has resistance which causes loss. The resistance increases
> approximately as the square root of the frequency due to the "skin
> effect". So, yes the copper in a sense becomes opaque. The energy in
> the signal gets dissippated in the copper and doesn't get to the load.
> We in electric land have have nothing with the low loss, wide band
> capabilities of a nice graded index fiber.

Ouch Del!

'nice _graded_ index fiber'?

A graded index fiber usually means a big (~50 um?) multimode fiber
employing a near parabolic refractory index profile to minimize
multi-mode dispersion.

However, even doing this leaves you with an order of magnitude less
bandwidth than a conceptually much simpler (step-index) single-mode
fiber afaik. (At least it did 24+ years ago when I wrote my thesis about
this kind of stuff!)

Terje

--
-
"almost all programming can be viewed as an exercise in caching"

From:	Del Cecchi
Subject:	Re: point-to-point busses
Date:	Thu, 20 Jan 2005 08:32:25 -0600

Terje Mathisen wrote:
> Del Cecchi wrote:
>
>> Copper has resistance which causes loss. The resistance increases
>> approximately as the square root of the frequency due to the "skin
>> effect". So, yes the copper in a sense becomes opaque. The energy in
>> the signal gets dissippated in the copper and doesn't get to the load.
>> We in electric land have have nothing with the low loss, wide band
>> capabilities of a nice graded index fiber.
>
>
> Ouch Del!
>
> 'nice _graded_ index fiber'?
>
> A graded index fiber usually means a big (~50 um?) multimode fiber
> employing a near parabolic refractory index profile to minimize
> multi-mode dispersion.
>
> However, even doing this leaves you with an order of magnitude less
> bandwidth than a conceptually much simpler (step-index) single-mode
> fiber afaik. (At least it did 24+ years ago when I wrote my thesis about
> this kind of stuff!)
>
> Terje
>

You prompted me to go do some review. You are indeed correct, graded
index is used to reduce dispersion in multimode fibres. And single mode
fibers are step index. Well, sort of. Apparently all sorts of stuff is
now being done with the cladding to reduce dispersion type effects in
single mode fibre as well. But indeed I was wrong to refer to graded
index single mode fiber. Thanks for the correction

And something I learned a while back that really surprised me is that
single mode fibre is significantly cheaper per meter than multimode
fiber is.

And both of them have way less loss and ISI than a copper wire. :-(

del cecchi

From:	Terje Mathisen
Subject:	Re: point-to-point busses
Date:	Thu, 20 Jan 2005 16:19:40 +0100

Del Cecchi wrote:

> Terje Mathisen wrote:
>> A graded index fiber usually means a big (~50 um?) multimode fiber
>> employing a near parabolic refractory index profile to minimize
>> multi-mode dispersion.
>>
>> However, even doing this leaves you with an order of magnitude less
>> bandwidth than a conceptually much simpler (step-index) single-mode
>> fiber afaik. (At least it did 24+ years ago when I wrote my thesis
>> about this kind of stuff!)
>>
>> Terje
>
> You prompted me to go do some review. You are indeed correct, graded
> index is used to reduce dispersion in multimode fibres. And single mode
> fibers are step index. Well, sort of. Apparently all sorts of stuff is
> now being done with the cladding to reduce dispersion type effects in
> single mode fibre as well. But indeed I was wrong to refer to graded
> index single mode fiber. Thanks for the correction
>
> And something I learned a while back that really surprised me is that
> single mode fibre is significantly cheaper per meter than multimode
> fiber is.

Not really surprising, since (a) you need much less high-quality raw
material (optical fiber is 3-4 orders of magnitude clearer than the best
glass used in expensive optical lenses), (b) you can make do with a
relatively coarse step between the core and the cladding.
>
> And both of them have way less loss and ISI than a copper wire. :-(

The _only_ real cost of fiber is at the ends.

Terje

--
-
"almost all programming can be viewed as an exercise in caching"

From:	Terje Mathisen
Subject:	Re: point-to-point busses
Date:	Wed, 19 Jan 2005 08:21:49 +0100

del cecchi wrote:
> Fanciest coding in the world can't beat shannon. And you better have
> enough high frequency capacity. Remember, the more you modulate the
> signal, the wider the bandwidth. Ahh, bessel functions. I remember
> them well.

The fanciest coding in the world would currently be a turbo style
infinite impulse response feedback code, right?

This approach can get you arbitrarily close to Shannon's limit, but at
the cost of a coding/decoding latency that also goes towards infinity.

Since you obviously know this, you probably meant something like getting
into the neighborhood of Shannon, right? :-)

Terje

--
-
"almost all programming can be viewed as an exercise in caching"

From:	Del Cecchi
Subject:	Re: point-to-point busses
Date:	Wed, 19 Jan 2005 14:40:52 -0600

Terje Mathisen wrote:
> del cecchi wrote:
>
>> Fanciest coding in the world can't beat shannon. And you better have
>> enough high frequency capacity. Remember, the more you modulate the
>> signal, the wider the bandwidth. Ahh, bessel functions. I remember
>> them well.
>
>
> The fanciest coding in the world would currently be a turbo style
> infinite impulse response feedback code, right?
>
> This approach can get you arbitrarily close to Shannon's limit, but at
> the cost of a coding/decoding latency that also goes towards infinity.
>
> Since you obviously know this, you probably meant something like getting
> into the neighborhood of Shannon, right? :-)
>
> Terje
>
yep, can't beat shannon. In fact you probably can't even tie shannon.
Of course NRZ is like 20dB worse than Shannon. :-( But it has low
latency :-)

Now about that 100 Gbit/second bandwidth....... That's a hell of a channel.

del cecchi

From:	already5chosen at yahoo.com
Subject:	Re: point-to-point busses
Date:	19 Jan 2005 15:26:11 -0800

I meant theoretical limit at unlimited bandwidth in the presence of the
thermal noise only.
Shennon says: C = W * log2(1+S/N)
Nyquist says: N = W * 4kT
where W=bandwidth
For very big W: S/N << 1
C = (W * S)/(N*ln2) = (W * S)/(W *4kT*ln2) = S/(4kT*ln2);
For S = 0.01W, T=300K
C = 0.87E18 b/s
In my previous post I was pessimistic by factor of 87.

From:	del cecchi
Subject:	Re: point-to-point busses
Date:	Wed, 19 Jan 2005 19:50:53 -0600

wrote in message
news:1106177171.769323.173230@f14g2000cwb.googlegroups.com...
> I meant theoretical limit at unlimited bandwidth in the presence of
the
> thermal noise only.
> Shennon says: C = W * log2(1+S/N)
> Nyquist says: N = W * 4kT
> where W=bandwidth
> For very big W: S/N << 1
> C = (W * S)/(N*ln2) = (W * S)/(W *4kT*ln2) = S/(4kT*ln2);
> For S = 0.01W, T=300K
> C = 0.87E18 b/s
> In my previous post I was pessimistic by factor of 87.
>
Not for any chip I worked on. I think you left out a few noise sources.

Shot noise for one, not counting man made noise sources.

del cecchi

From:	glen herrmannsfeldt
Subject:	Re: point-to-point busses
Date:	Mon, 10 Jan 2005 22:25:09 -0800

Mark W Brehob wrote:

(snip)

> Here is the question:
> What is the expected trend in bits/second/wire over a point-to-point
> connection (off-chip)? Do we expect to see a cap of just over a GHz or
> will see a steady increase? And if an increase, where do we think the
> number will be in 5 years?

> Related:
> What limits point-to-point bandwidth per pin? It it power, cost, or just
> raw physical limits? Something else (transistor switching speed?)

I don't know the real answer, but pin inductance has been a problem
for a long time. Ground bounce due to the inductance of ground
pins and such.

I would guess that pin inductance would eventually limit
the pin bandwidth. Also driving current decreases as transistors
get faster.

-- glen

From:	Del Cecchi
Subject:	Re: point-to-point busses
Date:	Tue, 11 Jan 2005 08:25:59 -0600

glen herrmannsfeldt wrote:
> Mark W Brehob wrote:
>
> (snip)
>
>> Here is the question: What is the expected trend in
>> bits/second/wire over a point-to-point
>> connection (off-chip)? Do we expect to see a cap of just over a
>> GHz or
>> will see a steady increase? And if an increase, where do we think the
>> number will be in 5 years?
>
>
>> Related:
>> What limits point-to-point bandwidth per pin? It it power, cost,
>> or just
>> raw physical limits? Something else (transistor switching speed?)
>
>
> I don't know the real answer, but pin inductance has been a problem
> for a long time. Ground bounce due to the inductance of ground
> pins and such.
>
> I would guess that pin inductance would eventually limit
> the pin bandwidth. Also driving current decreases as transistors
> get faster.
>
> -- glen
>
Most if not all high speed interfaces use small signal differential
signaling so the problem of SSO is lessened. And modern BGA packages
have quite low inductance.

And driving current doesn't decrease as transistors get faster, it goes
up which allows the same current to be driven by smaller transistors.

del cecchi

From:	David Kanter
Subject:	Re: point-to-point busses
Date:	11 Jan 2005 12:08:05 -0800

del and Paul,

There are PCs with PCIe ports. I have an Asus NCT-D, that has both
PCIe graphics and PCIe slots. There are no PCIe motherboards that
support the K8 that I know of. However, if you are happy with a
Prescott, the NCT-D should work, as will many others.

David Kanter

From:	already5chosen at yahoo.com
Subject:	Re: point-to-point busses
Date:	19 Jan 2005 11:54:09 -0800

del cecchi wrote:
> wrote in message
> news:1106068278.837190.102020@c13g2000cwb.googlegroups.com...
>
> Fanciest coding in the world can't beat shannon.

Yes, and the Shannon limit for a 10mW signal at 300K is what? About
10,000Tb/s?

From:	Del Cecchi
Subject:	Re: point-to-point busses
Date:	Wed, 19 Jan 2005 15:22:42 -0600

already5chosen@yahoo.com wrote:
> del cecchi wrote:
>
>> wrote in message
>>news:1106068278.837190.102020@c13g2000cwb.googlegroups.com...
>>
>>Fanciest coding in the world can't beat shannon.
>
>
> Yes, and the Shannon limit for a 10mW signal at 300K is what? About
> 10,000Tb/s?
>
Way smaller than that kemo sabe. What kind of channel are you assuming?
If it is a copper wire with 100 ohm impedence 10 mw is about 300 mv
RMS if I did my calculation right, and to get 10**16b/sec over a useful
distance one would have to have a signal to noise of what, 10**6 at
least. So I have to have an input stage with 300 nanovolt sensitivity,
and on a chip since we were talking about pin electronics. And it has
to have a bandwidth of 10GHz. Seems way unlikely.

del cecchi

From:	David Kanter
Subject:	Re: point-to-point busses
Date:	11 Jan 2005 12:06:30 -0800

Stephen Fuld wrote:
> "del cecchi" wrote in message
> news:34h03jF49t3vaU1@individual.net...
> >
> > "Paul Rubin" wrote in message
> > news:7xk6qlz57p.fsf@ruckus.brouhaha.com...
> >> Del Cecchi writes:
> >> > 2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
> >> > Ethernet CX4, and whatever they call double speed fibre channel)
> >>
> >> PCI-express is an interconnect for up to 5 meters? I'd thought it
was
> >> for plug-in cards on a backplane. Hmmm. I've been wondering what
low
> >> cost alternatives there are for low-latency RDMA between computers
in
> >> a cluster. The raw bandwidth isn't as important as speed of small
> >> transfers. Any advice?
> >
> > Yep, roll your own. Xilinx sells several FPGAs containing high
speed
> > serial I/O physical interfaces. What is it you want to attach to
what?
> > If it is X86, the best current solution that I know would be an
opteron
> > with HTX and InfiniPath from Pathscale, except you can't get it
quite
> > yet. It is still in "early availability" mode according to their
> > website. And I don't know about the availability of HTX on
Motherboards
> > either.
> >
> > Otherwise, does anyone make PC with PCI-Express ports, except for
> > graphics? Or one of the IB vendors as I mentioned before should be
> > affordable.

> Questions out of ignorance. Is a PCI Express port for graphics and
> different from any other PCI express port? Since the OP talked about
a
> database server, he probably doesn't need much graphics so could he
use any
> old graphics thing and use the PCI Express port for his
interconnect???
> Also, I thought that you could connect two PCI Express ports together
(on
> different motherboards) for a high speed interconnect, but to have
more than
> two, you needed the Advanced Switching extensions, which aren't quite
cooked
> yet. Is that right?

I believe that the PCIe graphics ports are different. They certainly
have a different physical connector (I have a system with both PCIe and
PCIe graphics).

I would stick with PCs that have PCIe...

David

From:	FredK
Subject:	Re: point-to-point busses
Date:	Tue, 11 Jan 2005 20:49:37 GMT

"David Kanter" wrote in message
news:1105473990.050919.90800@f14g2000cwb.googlegroups.com...
>

>
> I believe that the PCIe graphics ports are different. They certainly
> have a different physical connector (I have a system with both PCIe and
> PCIe graphics).
>
> I would stick with PCs that have PCIe...
>

PCI-express is based on 1 or more "lanes" (up to 32 I think),
where the lane is a bi-directional serial link. At the slowest
simplest would be one lane. Using 2 lanes gets you twice the
bandwidth.

The standard connectors are 4, 8 and 16 lanes. So far only
graphics and some high-end cluster interconnects are using
16-lanes. Cards and slots are supposed to play such that you
can put a card with fewer lane requirements into a slot with
more lanes, but each slot "type" was supposed to provide all
the lanes defined for it. But of course, they also started with
an exception and allow an 8 lane slot to be underfunded with
4 lanes. They are now talking about allowing 16 lane slots to
be underfunded as well.

From:	Nick Maclaren
Subject:	Re: point-to-point busses
Date:	10 Jan 2005 10:42:26 GMT

In article ,
Mark W Brehob writes:
|>
|> At the University of Michigan we just had Robert Colwell (formally of Intel)
|> come in and give a talk. After the talk one issue came up which I found I
|> was disagreeing with everyone about (generally a sign I'm wrong of course).

Who said what?

|> Here is the question:
|> What is the expected trend in bits/second/wire over a point-to-point
|> connection (off-chip)? Do we expect to see a cap of just over a GHz or
|> will see a steady increase? And if an increase, where do we think the
|> number will be in 5 years?

An increase, but not necessarily a steady one. We are hitting some
physical limits, and don't know how many more can be resolved. No,
I can't explain what they are in detail, being a mediocre physicist.

|> Related:
|> What limits point-to-point bandwidth per pin? It it power, cost, or just
|> raw physical limits? Something else (transistor switching speed?)

Physical limits, but not raw ones.

For copper (electrical): leakage, crosstalk and noise generally.
The 10GE people are heavily into this.

For fibre (optical): making fast, integrated lasers for conversion.
We aren't yet at the fibre limits, though they will hit eventually.

|> What does the cost per signal pin look like? Put another way, at a
|> reasonable price, what is the bandwidth trend for point-to-point
|> communication?

Erratically upwards.

|> I did a bit of a paper search (~1 hour) and turned up very little. Hot
|> Interconnects seemed to have a fair bit, but no real trends or predictions I
|> could find.

For the reasons I give above. Every development needs solutions to
some nasty problems.

Regards,
Nick Maclaren.

From:	Kai Harrekilde-Petersen
Subject:	Re: point-to-point busses
Date:	Mon, 10 Jan 2005 16:38:35 +0100

nmm1@cus.cam.ac.uk (Nick Maclaren) writes:

> In article ,
> Mark W Brehob writes:
> |>
> |> At the University of Michigan we just had Robert Colwell (formally of Intel)
> |> come in and give a talk. After the talk one issue came up which I found I
> |> was disagreeing with everyone about (generally a sign I'm wrong of course).
>
> Who said what?
>
> |> Here is the question:
> |> What is the expected trend in bits/second/wire over a point-to-point
> |> connection (off-chip)? Do we expect to see a cap of just over a GHz or
> |> will see a steady increase? And if an increase, where do we think the
> |> number will be in 5 years?

3.125Gbit/s (for XAUI / 10G Ethernet) is already in reasonably wide
deployment and 10.3Gb/s (XFI) is coming along.

> An increase, but not necessarily a steady one. We are hitting some
> physical limits, and don't know how many more can be resolved. No,
> I can't explain what they are in detail, being a mediocre physicist.
>
> |> Related:
> |> What limits point-to-point bandwidth per pin? It it power, cost, or just
> |> raw physical limits? Something else (transistor switching speed?)
>
> Physical limits, but not raw ones.
>
> For copper (electrical): leakage, crosstalk and noise generally.
> The 10GE people are heavily into this.

Errh, it's not the copper per see, that is the problem. It's the loss
of the crappy FR-4 PCB material that is the problem. In fact, cables
will quite often have better worst-case parameters than standard FR-4.

You can run 10-11Gbit/s (on a differential pair) for a short distance,
but don't put too many connectors, via transitions etc into that path
please. I'd limit it to less than 10 cm (4 inches).

I walked down the hall and talked to our resident high-speed guru, and
he said that even at 1G the loss-effects can be seen. He wouldn't
says whether the skin-effect loss or the dielectric loss is dominant,
as it depends too much on the actual PCB fabricated. Due to the small
feature sizes these days, a differential strip-line pair can see pure
resin in the horizontal plane and mostly glass in the vertical,
yielding two different propagation delays in the X and Y axis,
resulting in non-TEM waves.

> |> What does the cost per signal pin look like? Put another way, at a
> |> reasonable price, what is the bandwidth trend for point-to-point
> |> communication?

Depends on your perspective. Having pre-emphasis, equalizing, SERDES,
CMU, PLLs, encoding (apparently 8B/10B is everyones favorite these
days), and protocol machine can add quite a burden on a chip.

Regards,

Kai
--
Kai Harrekilde-Petersen

From:	Nick Maclaren
Subject:	Re: point-to-point busses
Date:	10 Jan 2005 16:18:00 GMT

In article ,
Kai Harrekilde-Petersen writes:
|> >
|> > |> Related:
|> > |> What limits point-to-point bandwidth per pin? It it power, cost, or just
|> > |> raw physical limits? Something else (transistor switching speed?)
|> >
|> > Physical limits, but not raw ones.
|> >
|> > For copper (electrical): leakage, crosstalk and noise generally.
|> > The 10GE people are heavily into this.
|>
|> Errh, it's not the copper per see, that is the problem. It's the loss
|> of the crappy FR-4 PCB material that is the problem. In fact, cables
|> will quite often have better worst-case parameters than standard FR-4.

Well, I said that I didn't know the details :-)

The sort of complications you refer to are why I don't believe that
we will see a steady trend. Every time the speed is ramped a notch,
a new set of problems will stick up and have to be hammered down
again. Some will be easy to deal with; others won't.

Regards,
Nick Maclaren.

From:	Del Cecchi
Subject:	Re: point-to-point busses
Date:	Mon, 10 Jan 2005 08:58:29 -0600

Mark W Brehob wrote:
> Hello all,
>
> At the University of Michigan we just had Robert Colwell (formally of Intel)
> come in and give a talk. After the talk one issue came up which I found I
> was disagreeing with everyone about (generally a sign I'm wrong of course).
>
> Here is the question:

OK, right down my alley. Free consulting. Such a deal.

> What is the expected trend in bits/second/wire over a point-to-point
> connection (off-chip)? Do we expect to see a cap of just over a GHz or
> will see a steady increase? And if an increase, where do we think the
> number will be in 5 years?

For moderate length interconnections (up to maybe 5 meters)

1 Gbit/second/pair is old news. Fibre Channel was at this rate a while
back. (1.0625 Gb/s)

2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express, Ethernet
CX4, and whatever they call double speed fibre channel)

5 Gb/second is coming soon. InfiniBand DDR, PCI-Express Gen 2, CEI from
OIF.

10 Gbit is coming not far behind.

>
> Related:
> What limits point-to-point bandwidth per pin? It it power, cost, or just
> raw physical limits? Something else (transistor switching speed?)

All of the above. Jitter, high frequency attenuation, gain bandwidth,
noise, materials, crosstalk, etc. Which one is the most significant
depends on the objectives, like distance and number of connections.
>
> What does the cost per signal pin look like? Put another way, at a
> reasonable price, what is the bandwidth trend for point-to-point
> communication?

Just the PHY level on an asic is pretty cheap and widely available now
at 2.5 Gbits (HyperTransport style or Serial) 5 Gb/s is coming
relatively soon. And really won't cost much more. But may be priced
higher. The trend is up. :-)
>
> I did a bit of a paper search (~1 hour) and turned up very little. Hot
> Interconnects seemed to have a fair bit, but no real trends or predictions I
> could find.
>
> Mark

This is my specialty, more or less.

del cecchi
>

From:	Paul Rubin
Subject:	Re: point-to-point busses
Date:	10 Jan 2005 07:14:34 -0800

Del Cecchi writes:
> 2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
> Ethernet CX4, and whatever they call double speed fibre channel)

PCI-express is an interconnect for up to 5 meters? I'd thought it was
for plug-in cards on a backplane. Hmmm. I've been wondering what low
cost alternatives there are for low-latency RDMA between computers in
a cluster. The raw bandwidth isn't as important as speed of small
transfers. Any advice?

From:	del cecchi
Subject:	Re: point-to-point busses
Date:	Mon, 10 Jan 2005 21:39:55 -0600

"Paul Rubin" wrote in message
news:7xk6qlz57p.fsf@ruckus.brouhaha.com...
> Del Cecchi writes:
> > 2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
> > Ethernet CX4, and whatever they call double speed fibre channel)
>
> PCI-express is an interconnect for up to 5 meters? I'd thought it was
> for plug-in cards on a backplane. Hmmm. I've been wondering what low
> cost alternatives there are for low-latency RDMA between computers in
> a cluster. The raw bandwidth isn't as important as speed of small
> transfers. Any advice?

Yep, roll your own. Xilinx sells several FPGAs containing high speed
serial I/O physical interfaces. What is it you want to attach to what?
If it is X86, the best current solution that I know would be an opteron
with HTX and InfiniPath from Pathscale, except you can't get it quite
yet. It is still in "early availability" mode according to their
website. And I don't know about the availability of HTX on Motherboards
either.

Otherwise, does anyone make PC with PCI-Express ports, except for
graphics? Or one of the IB vendors as I mentioned before should be
affordable.

del cecchi

From:	Stephen Fuld
Subject:	Re: point-to-point busses
Date:	Tue, 11 Jan 2005 06:06:25 GMT

"del cecchi" wrote in message
news:34h03jF49t3vaU1@individual.net...
>
> "Paul Rubin" wrote in message
> news:7xk6qlz57p.fsf@ruckus.brouhaha.com...
>> Del Cecchi writes:
>> > 2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
>> > Ethernet CX4, and whatever they call double speed fibre channel)
>>
>> PCI-express is an interconnect for up to 5 meters? I'd thought it was
>> for plug-in cards on a backplane. Hmmm. I've been wondering what low
>> cost alternatives there are for low-latency RDMA between computers in
>> a cluster. The raw bandwidth isn't as important as speed of small
>> transfers. Any advice?
>
> Yep, roll your own. Xilinx sells several FPGAs containing high speed
> serial I/O physical interfaces. What is it you want to attach to what?
> If it is X86, the best current solution that I know would be an opteron
> with HTX and InfiniPath from Pathscale, except you can't get it quite
> yet. It is still in "early availability" mode according to their
> website. And I don't know about the availability of HTX on Motherboards
> either.
>
> Otherwise, does anyone make PC with PCI-Express ports, except for
> graphics? Or one of the IB vendors as I mentioned before should be
> affordable.

Questions out of ignorance. Is a PCI Express port for graphics and
different from any other PCI express port? Since the OP talked about a
database server, he probably doesn't need much graphics so could he use any
old graphics thing and use the PCI Express port for his interconnect???
Also, I thought that you could connect two PCI Express ports together (on
different motherboards) for a high speed interconnect, but to have more than
two, you needed the Advanced Switching extensions, which aren't quite cooked
yet. Is that right?

--
- Stephen Fuld
e-mail address disguised to prevent spam

From:	Paul Rubin
Subject:	Re: point-to-point busses
Date:	10 Jan 2005 22:26:15 -0800

"del cecchi" writes:
> > a cluster. The raw bandwidth isn't as important as speed of small
> > transfers. Any advice?
>
> Yep, roll your own. Xilinx sells several FPGAs containing high speed
> serial I/O physical interfaces. What is it you want to attach to what?
> If it is X86, the best current solution that I know would be an opteron
> with HTX and InfiniPath from Pathscale, except you can't get it quite
> yet. It is still in "early availability" mode according to their
> website. And I don't know about the availability of HTX on Motherboards
> either.

That sounds promising, yes, I'm interested in the general problem of
clustering with cheap commodity hardware, more for database-like
servers than for number crunching. That does tend to mean x86.

Infiniband adapters seem to start around $1K per node, which is orders
of magnitude more expensive than Firewire or 1G Ethernet. Switches
seem to start around $10K. Myrinet is also around $1K per node.
It's possible that I'm missing something, though.

Maybe I should just think about kernel software hacks to use low-level
Ethernet packets for RDMA, bypassing TCP/IP, using x86-64 paging
hardware to map the memory of a whole cluster into each node's user
space, and handling page faults over the switched ethernet. I've also
heard of such a thing as low latency 1G ethernet cards. Is that some
special type of card with a nonstandard protocol, or just some
particular card that happens to do well on small-packet benchmarks?

Is there a better newsgroup for this kind of question?

And on a more traditional comp.arch subject, the x86-64 gets its huge
address space by having something like five levels of page tables. Is
that workable at all?! I'd never heard of other 64 bit cpu's doing
that.

thanks

From:	Paul Rubin
Subject:	Re: point-to-point busses
Date:	10 Jan 2005 22:28:04 -0800

Paul Rubin writes:
> That sounds promising, yes, I'm interested in the general problem of
> clustering with cheap commodity hardware, more for database-like
> servers than for number crunching. That does tend to mean x86.

Oh shoot, I should have put this in the firewire thread that I started.
I didn't mean to hijack the point-to-point bus thread. Sorry.

From:	Del Cecchi
Subject:	Re: point-to-point busses
Date:	Mon, 10 Jan 2005 13:28:32 -0600

Paul Rubin wrote:
> Del Cecchi writes:
>
>>2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
>>Ethernet CX4, and whatever they call double speed fibre channel)
>
>
> PCI-express is an interconnect for up to 5 meters? I'd thought it was
> for plug-in cards on a backplane. Hmmm. I've been wondering what low
> cost alternatives there are for low-latency RDMA between computers in
> a cluster. The raw bandwidth isn't as important as speed of small
> transfers. Any advice?

If it isn't 5 meters it soon will be. It isn't the physical, it's the
logical stuff for switches et al.

What does the InfiniBand stuff cost? The switches seem reasonable
unless it is personal money. 4X stuff from one of the three hardware
companies would do the job. Plugs into PCI-X.

Or there is the HTX to IB that plugs into hypertransport. Pathscale,
right Greg? Should be motherboards with HTX connectors out Real Soon
Now. http://www.pathscale.com/pdf/PathScaleInfiniPath.pdf

In fact that seems the most attractive solution if the price is
reasonable. Couldn't get a price on the website.

del cecchi (clearly personal opinions)

del cecchi

del cecchi