10 GB Ethernet is expensive due to low volumes. Fiber channel is lower cost, but still not high volume and not cheap enough for mass deployments. Now that Apple and Intel have announced Thunderbolt, 10 GB IO connections will be low cost.

Why not use Thunderbolt for SAN and network connectivity. Look at the difference between these two designs.

Figure 1 illustrates a typical topology of building out a server cluster today, in which, while the form factors may change, the basic configuration follows a similar pattern. Given the widespread availability of open-source software and off-the-shelf hardware, companies have successfully built large topologies for their internal cloud infrastructure using this architecture.

Figure 1: Typical Data Center I/O interconnect

Figure 2 illustrates a server cluster built using a native PCIe fabric. As is evident, the usage of numerous adapters and controllers is significantly reduced and this results in a tremendous reduction in power and cost of the overall platform, while delivering better performance in terms of lower latency and higher throughput.

Figure 2: PCI Express-based Server Cluster

We'll see what server vendor is first with Thunderbolt support. For now some innovative users could use a bunch of MacBook Pros.