Maximum PC

Actually I have all of the systems sharing a network folder where the client executable resides. I have a couple of batch files that I use to boot 8 clients on a machine. Point is, updating the client is replacing the file in one place.

Jason

Wow.

I'd love to know how you did this/have it set up.
Even with only 16 clients it gets old quick
doing it the mortal way.

UPDATE 7-16-2010: Added a couple of lines to GPURUN.BAT that allow each GPU client to automatically copy the new DLLs required by GPU3 into their local directories.



Smoke, mirrors, batch files, and the GPU Console Client (not Windows Tray Client).

I have a network share on one of the computers, node Dagny since it was the first one I built. Looks like this:



As I add a client machine I name it and add a corresponding folder on node Dagny under C:\FAH\MACHINES\. On each of the clients I map \\DAGNY\FAH as network drive F: (including on node DAGNY where the share resides, you'll see why in a sec).

Next is a bit of secret sauce. At some point I did a basic GPU client install with -configonly for each of 8 GPUs to specify their team, machine ID, etc. But I copied the 8 directories off to a USB stick before I ever ran the clients... This gives me 8 virgin GPU folders GPU0-GPU7 with just the basic cudart.dll, client.cfg and whatnot. I copy those virgin GPU client folders into Dagny's C:\FAH\MACHINES\NEWNODENAME\ folder. Still with me? Now I have C:\FAH\MACHINES\NEWNODENAME\GPU0 through C:\FAH\MACHINES\NEWNODENAME\GPU7 sitting there ready to fire up a client for the first time, make the WORK directory, download the fah_core, download work, etc.

In Dagny's C:\FAH folder (the "root" of the F: share) I have a simple batch file called RUN8.BAT (I also have RUN1, RUN2, RUN4, RUN6, etc.):



Now, in the GPU folder under the share I have the GPU F@H client renamed "GPU.EXE" and its DLLs along with a simple batch file called GPURUN.BAT:



See, that batch file uses the %USERDOMAIN% command shell macro to move to the correct MACHINE directory on the network share and then the 0-7 command line variable from the first batch file to switch to the correct GPU# directory on the network share.

F:\MACHINES\%USERDOMAIN%\GPU%1
  becomes something like
F:\MACHINES\REARDEN\GPU3

I create a shortcut to the F:\RUN8.BAT batch file on the desktop of the new client. Now when I click on the shortcut it runs the batch file with all the START commands in it which starts 8 shells by calling the second batch file 8 times. The second batch file changes to the correct directory for that specific GPU client, specifies all of the command line parameters for the client and adds a nifty title to the CMD window so I can easily tell the clients apart at a glance (i.e. "REARDEN - GPU 3")

Notice though that there are extra batch file command line macros %2-%6 after the standard command line. These additional command line options will be empty (null) if the RUN8.BAT batch file doesn't use them and therefore have no effect, but if you want to have some GPUs run with additional options you can copy the RUN8.BAT file instead of creating a shortcut to it. Then in the copy of the batch file on your desktop you can add additional command line options to customize the behavior of some GPU clients to be different than the default. This gives a bit more flexibility to the system -- not every client must run the same command line.

Since every single GPU client ultimately uses GPURUN.BAT to start, I can change every client's command line in one central location. Since every GPU client uses the same GPU.EXE to run, I can update the client software by replacing one executable file.

I arrived at this solution after much gnashing of teeth trying to keep the 34 clients in the rack coordinated. Originally I had discrete setups on each node's drive, but I was making many errors trying to manually keep them synched.

Here's the neat part. When I add a new node the only thing I have to do to the local machine is map \\DAGNY\FAH to F:, copy a shortcut to F:\RUN8.BAT to the desktop and double click it. That's IT. There is no F@H software on the client machines unless you want to run SMP.

In summary the entire node addition procedure is:
1. Go to node Dagny, C:\FAH\MACHINES
2. mkdir NEWNODENAME
3. Copy GPU0-GPU7 virgin dirs from USB stick to C:\FAH\MACHINES\NEWNODENAME
4. Go to new machine, map \\DAGNY\FAH to F:
5. Create shortcut to F:\RUN8.BAT on new client's desktop
6. Click shortcut and watch 8 console GPU clients spring up

Finished product looks like this (click to enlarge):



Jason

Wow again.

That was way too much for my little head to digest.

Do the machines have to be on a domain, or can they be on a home network 'workgroup'?

Mine are all in WORKGROUP. Doesn't matter as long as you can map the share.

Jason

if i may be sincerely direct here...

THAT'S JUST FUCKING AMAZING... GREAT JOB...

you not only answered my question but about 20 others that haven't even occurred to me yet.... thanks for the info.... it's fun watching how people work with the f@home clients.... if i may ask as well what do you do for a living...

very impressive mang...

staypuff,

Necessity is the mother of invention. It was all fun and games until I added nodes 3 and 4. Then just logging in to each client and double clicking 8 start icons was getting ridiculous. I had to come up with a solution. I still have to log in to each machine to start the 8 clients since the batch file must be executed in the context of the machine running the clients, but now it's just logging in, clicking one icon and logging out.

About OS's, the best OS for GPU folding on GTX295s or 9800GX2s is Windows XP 32-bit or 64-bit. Vista CUDA is bugged at detecting the second GPU in these dual GPU cards (unless it's been fixed very recently). Some folks have intermittently gotten Vista to work with dual GPU cards but they're the minority.

Another compelling benefit of XP is that you do not need dummy plugs to fake a monitor on each of the cards' outputs as you do with Vista. In Vista you cannot extend the desktop onto a given output unless a monitor is detected. This is a Microsoft WHQL requirement, and while nVidia could program around it, it would mean that their driver could not receive WHQL certification. I've seen some people writing about a method around needing dummy plugs in Vista but I honestly haven't paid much attention since the issue doesn't affect me any longer.

Jason

thanks Jason... i've been with XP on everything so far and it's been extremely easy to setup on this os.... my "soon to be here" 295 will be on XP.... i've noticed this on the Stanford Folding forums that most of the people that were claiming success were using XP....

Excellent job by yourself to streamline the process... you are to be congratulated.... going to extremes in projects like this is honorable...

you sure you aren't a monkey????

we have room in our clubhouse???

oh just one other thing.... any idea why the flag -g80 has to be added at the end???? i've seen some that say you don't have to add it and others that say you must... considering this isn't remotely a g80 gpu any idea why this flag is needed at all???? really curious as there seems to be differences in opinion out there in folding land... i'm going to add this flag myself as you've had obviously good success...

You shouldn't need it on Windows. It's more there for the Linux/GPU folders so that the GPU detection always passes.

ok.... i'm just curious as to why everyone is using these flags.... i'm also curious as to why Stanford hasn't said a fucking thing about it one way or the other... you'd think they'd try to come up with a proper guide..... it's like a gaming company asking you to pay for the game then then telling you to code it yourself..... i've seen 3 different ways to add flags for dual gpus and some work and some don't.....

i'll try both ways when the card comes in... you'd think that people that donate time and money to a project wouldn't have to figure it out for themselves however....

I used to never use "-forcegpu nvidia_g80". However, when building one of my machines I spent a couple of hours once trying to figure out why the GPU client kept telling me that my GPU wasn't supported. It was behaving as though SLI was still engaged even though it definitely wasn't. Eventually I tried this flag even though I'd never needed it before and it fixed the problem. So now I just include it all the time.

Jason

Dude

We are not worthy dude

heh my x1900xtx burned up rendering that pic in jealousy

I've seen people who've had trouble with their clients - after looking at their log someone points out that the client thinks they have ATI card. They add the flag and problem goes away.
I didn't need it on my first multi GPU rig. But I use it all the time now. Even though your card isn't really a g80, the g80 is the only recognized parameter for the "-forcegpu" flag by the client. I've seen people SAY they use g92, but g92 doesn't actually exist to the client. There are only the nvidia_g80, then two different ATI processors.

Don't know if that helps clear up some of the other recommendations you've seen puffy?

So, now you consider yourself a people? Eh, monkey?

Jason,

Great work. Could you share more specifics about the 2x750W and the cable?

I know Jaak (Estonia Donates) had difficulties with PSUs also.

Thanks,

Denny

Now...not to seem obtuse for making the suggestion...but, you have your mission critical drive in a RAID 1, no? Or a RAID 5?

Seems to be definitely the place for such a thing...especially since we're talking about a very high concentration of heat, and relatively high IOs...

Actually, seems like a good place to put a solid state disk, now that I think about it, but, again, don't want to seem to mock your hardware, or your purpose for doing it...

It all seems so...noble. In an awesomely nerdy way.

Hey guys,

@CPUacreage:
In my first two shelves I used a single 1250W Cooler Master PSU. This power supply can accommodate 4 cards with 4 4-pin and 4 8-pin PCIe power connectors. I have not had a single problem running those entire machines on those supplies, but since the power draw of the system is so high (~1140W with SMP running) and those supplies do not have a single 12V rail there is no question that I'm pressing them fairly hard. Also, they got "deactivated" on newegg.com. That was alright though, because I'd been planning on moving to a dual supply solution anyhow.

Now I'm using a pair of PC Power & Cooling 750 80+ certified SLI PSUs. I wire those two supplies to switch together with Lian Li Dual Power Supply Adapter, plug 2 cards into one supply and two cards into the other and it just works. I also put the HDD on one and the motherboard on the other to even out the load a bit further. This has worked perfectly, I can't tell I'm using two supplies. I've now built two shelves this way.

Jaak and I have talked about this stuff a little bit in emails and on the F@H forums.

@Ilander
If the conversation is civil and constructive I don't mind at all talking about this stuff. You're neither mocking or obtuse; don't sweat it.

This stuff isn't exactly Mission Crititcal. It won't affect my ability to make a living or injure someone if it fails. Nevertheless, setting up a task to copy that primary disk's FAH directory tree off to one of the secondary machines at midnight or something isn't a bad idea.

There's nothing on the primary disk other than an OS, a few easily downloadable F@H files and the work units. If that disk died I would cuss it, hurl it into the trash, slap a new one in and reinstall Windows XP x64. And although folks talk about heat all the time when they see my system, it really isn't that hot, especially with the new 5900 series WUs which run much cooler.

I did consider solid state disks when I first started the project, especially when I saw their prices had dropped to under $100.00. But then I saw that the cheapest spinning drives were $34.00 for 80GB; can't beat that with a stick. I'm paying that much for the RAM, it's really amazing. I thought about trying to make the remote nodes diskless and do network booting, but at $34.00 a node for the disks it just not worth the hassle.

Hope that clarifies some stuff for you guys.

Jason

Actually, it wouldn't be a good place for SSD. SSD's have a significantly lower life expectancy that spinning platter HD's - in terms of the number of times a location on the drive can be written over. With so many GPU work folders turning clients around so quickly, you would kill that drive (or drives) in no time.

My $.02 on the matter.

Everything is all fine and dandy but.........for the most MaximumPC fun you need to run the 11108 "flag".