How to set up a cluster?

[perfetti]

Dear everyone,
       I am now using three computers(one workstation) in parallel, however, it's still not fast enough.
       We want to set up a cluster, but I don't know what do we need to set up a cluster(hardware,software)?
       Can we still use the three computers we have? We have windows on all the computers, and linux on the workstation.
       Thank you!
       

[Anders Blom]

Eh, yes, but I didn't get 100% what you have. You say 3 computers, 1 workstation... Which is it?

If you have 3 computers (boxes, machines), you can easily connect them in parallel via a hub, install MPICH2 on all machines, and run ATK over MPI on them. It takes only basic sysadm skills, and there are several tips on this Forum already, plus if you Google it.

[perfetti]

Yes we've already done that.
The workstation is one of the three computers.

Now we want to buy new computers to make it run much faster.
My advisor also want to set up a cluster.

Are the paralleled computers what you mean as cluster? 
I know there's something that needs red hack, linux machine, and special hardware, etc....
Would that be much different?

[Anders Blom]

Unless you are an expert, I recommend buying 4-5 Dell machines (or whatever brand you fancy), put Ubuntu Linux on them (simple to install, simple to maintain), then connect to a good hub, install MPICH2, and you have a cluster.

Get 8 GB of RAM per machine, some decent but not top-of-the-line CPU, and reasonable disks (standard config will be fine). No need for fancy graphics cards or DVD etc. These machines can be obtained for between 400 and 600 USD a piece, I think.

Cooling and noise will be a concern, hopefully you have a server room where you can place them.

[perfetti]

1. We are now using windows cause ATK doesn't run successfully on linux on one machine.
Will linux be much faster than windows?

2. We just connect them via internet, will a hub be better? What's the difference?

3. We need monitor for each of the computer right now, is there anyway that we don't need monitor for each machine but can still control over them?  Like installing software, copying file , and watching the performance to each machine.

[Anders Blom]

1. It's not a huge difference, but installing Windows on 5 machines will not be fun, to maintain etc, and struggle with ssh and so on. Go with Linux. ATK should run fine on any Linux, not sure what you experience was (tell me). VNL can be trickier, but ATK is pretty rock-solid.

2. If you have 5 internet ports in the room there is perhaps less of a difference, but why waste the 5 ports when you can use hub (or better, a switch!).

3. You would not buy monitors for each machine. If you desperately need to, just move the cable around, but 99% of all admin tasks can be done on the command line in Linux.

[perfetti]

3. You would not buy monitors for each machine. If you desperately need to, just move the cable around, but 99% of all admin tasks can be done on the command line in Linux.

Is it the same situation if I run jobs under windows?
Do I still need monitors for windows systems? It would be easier since usually the computers would have windows installed when sold.

[Anders Blom]

First of all, I would strongly recommend to go with a Linux cluster. While you certainly can get decent performance on Windows in parallel, it is a fact that unless you configure it specifically for HPC purposes, the operating system itself consumes so many resources that it leaves less for the calculation job, not least memory. In most of my experience I see about 1.5-2x performance in serial on Linux on equivalent machines (in serial already).

I understand you are reluctant to take on the administrative Linux tasks, and you can of course start with Windows (as you say, it comes with the machines, usually) and see what you think about the performance. But don't buy 5 monitors There are monitor switches for far less money.

Also, note that you will be working on the command line even in Windows, to run in parallel (for now, at least - we have interesting plans for ATK 12.8, to let you run on a cluster from within VNL!).