scalable storage solution using ZFS: OSOL / Nexenta?

some one commented that ZFS is not a distibuted file system and thus more suitable for storage from a box than a cluster.

is it possible to start small as NAS with Nexenta / opensolaris and scale up in future by adding more SAS JBODs?

if so, what needs to be taken care of at the hardware list? for better performance?

i mean what to choose for storage adapters, HBA etc

From serverfault maruti

• ZFS isn't a cluster file system, that's for sure, so can be used for single-server access but not for multiple. That said it makes a wonderful basis for a non-clustered NAS.

  As for the Nexenta question - yes you can scale it very well from small to very large quite easily.

  For performance I'd be tempted to stick to a single 4-6 core CPU, you won't benefit from many more cores. Try to get at least 4GB so you'll have a nice large cache. Use Dual teamed NICs. When picking controllers pick a name brand, you can't really go wrong with Adaptec, make sure it supports exposing all disks raw when in JBOD mode (i.e. if you have 16 drives does it appear as 16 drives, what's the most of these supported per controller). Always go SAS rather than SATA (although those new RE3 disks are quite nice and are genuine 24/365 drives too). Make sure you get dual power supplies. If possible use dual-port SAS disks and ensure both paths are live. Setup whatever alerting/monitoring you can with the controller/s to let you know IMMEDIATELY when you have a dead disk etc. Oh and buy at least one spare disk to be kept 'cold' and if you can have another set as 'hot' standby.

  Other than that get on with it :)

  maruti : hi, DL370G6 with Xeon 55xx, 16GB RAM + MDS 600 as JBOD. This JBOD has 46 15K drives...4 SAS ports. DL may have 4 1Gb NICs...do i need more for connecting 4 clients/consumers? can OSOL / Nexenta do hybrid provision? that is iscsi, NFS or CIFS from the same box?

  Chopper3 : Go for an E5630 if you can, it's a nice balance of cost/heat/performance/price. Don't do 16GB of memory, those chips don't work like that, they go up in 3 chips per physcial CPU at a time (i.e. for a single CPU box get 3 x 1GB, 3 x 2GB, 3 x 4GB etc. not 2 x something or 4 x something), 12GB will be fine. HP support that box with Solaris so that's good, all the drivers should be there - might need to check the drivers for the disk controller as none of the P212/P411/P812 officially support Solaris - which model are you going for? 370 does have 4 NICs, nic teaming is supported too so you're ok

  maruti : thanks for your valuable feedback.P410 with 512 cache. anything to worry about SAS interface cards on MDS600? also thinking of HP procurve 3500yl Gb switch-layer3. overall is this a nicely balanced ZFS based solution?

  Chopper3 : The P410 is an internal only card - you'll need an external one with the mini-SAS connectors to drive the 600. The 212/411 and 812 are the ones you're looking for. My only worry is the lack of official Solaris driver support for those three cards - might be work asking HP directly or doing a bit of googling. I know almost nothing of HP switches sorry, I'm a Cisco-man through and through when it comes to networking. Perhaps ask any questions about the HP as a different question?

  maruti : looks like HP P410 is based on PMC Sierra chipset. likely supported by opensolaris: http://opensolaris.org/jive/thread.jspa?messageID=458335

  Chopper3 : Great research maruti, I didn't know that.

  maruti : @Chopper. tried nexenta 3.0.3 on HP P410i.....it detected HP logical volume...next cleared the array volumes and exposed to nexenta, could not find any drives...no JBOD it seems, BTW it is OSOL b134
  From Chopper3

• i actually build these kinds of systems for a living.

  SuperMicro kit works great. Stay with Intel for NIC's. Use SSD's for ZIL (mirror this for gods sake) and L2ARC. Stripe RAIDz2 across the JBOD and the entire ARRAY to give you're self max safety. De-Dupe will need at least 24GB of RAM, use ECC DDR3 Registered and throw the money at this and a couple of 5660's if you can!

  You can indeed start small - something like a http://www.supermicro.com/products/chassis/3U/?chs=836 (you will need to buy mobo,mem and proc for this).

  Or take a look at http://www.va-technologies.com/sbb

  maruti : i cant go with SuperMicro, we have an agreement with HP and i have chosen DL370 G6 + MDS600 for a NAS solution based on ZFS/Nexenta. The MDS600 has 4 SAS ports...DL370 G6 has 4 NIC ports do i need more?

  Khushil : You connecting from the MSD6000 to the DL370 and then serving out from the 4NIC? 1GbE? Bond them for better throughput or go for 10GbE I would suggest. Make sure you have Intel NIC's as the broadcom support on OpenSolaris is - at the moment, it's been worked on - sub-par.
  From Khushil