Quantcast

Maximum PC

It is currently Mon Dec 22, 2014 1:34 am

All times are UTC - 8 hours




Post new topic This topic is locked, you cannot edit posts or make further replies.  [ 63 posts ]  Go to page Previous  1, 2, 3, 4  Next
Author Message
 Post subject:
PostPosted: Tue Mar 17, 2009 11:32 am 
8086
8086

Joined: Sat Mar 14, 2009 10:02 pm
Posts: 14
Quakindude wrote:
I bet you get a serious case of the ass when a F@H client expires. :twisted:


Actually I have all of the systems sharing a network folder where the client executable resides. I have a couple of batch files that I use to boot 8 clients on a machine. Point is, updating the client is replacing the file in one place.

Jason


Last edited by Atlas Folder on Tue Mar 17, 2009 7:21 pm, edited 1 time in total.

Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 12:28 pm 
Monkey Federation (Top 10)
Monkey Federation (Top 10)
User avatar

Joined: Tue Oct 11, 2005 5:20 pm
Posts: 10472
Location: Kansass
Atlas Folder wrote:
Quakindude wrote:
I bet you get a serious case of the ass when a F@H client expires. :twisted:

Actually I have all of the systems sharing a network folder where the client executable resides. I have a couple of batch files that I use to boot 8 clients on a machine. Point is, updating the client is replacing the file in one place.

Jason


Wow.

I'd love to know how you did this/have it set up.
Even with only 16 clients it gets old quick
doing it the mortal way.


Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 2:12 pm 
8086
8086

Joined: Sat Mar 14, 2009 10:02 pm
Posts: 14
UPDATE 7-16-2010: Added a couple of lines to GPURUN.BAT that allow each GPU client to automatically copy the new DLLs required by GPU3 into their local directories.

Quote:
Jason how the hell did you keep your sanity installing all those clients... since myself and a few others are either trying to get installed or in the process of receiving gtx295's can we ask what os's you're using and what flags or setup process you used???


Smoke, mirrors, batch files, and the GPU Console Client (not Windows Tray Client).

I have a network share on one of the computers, node Dagny since it was the first one I built. Looks like this:

Image

As I add a client machine I name it and add a corresponding folder on node Dagny under C:\FAH\MACHINES\. On each of the clients I map \\DAGNY\FAH as network drive F: (including on node DAGNY where the share resides, you'll see why in a sec).

Next is a bit of secret sauce. At some point I did a basic GPU client install with -configonly for each of 8 GPUs to specify their team, machine ID, etc. But I copied the 8 directories off to a USB stick before I ever ran the clients... This gives me 8 virgin GPU folders GPU0-GPU7 with just the basic cudart.dll, client.cfg and whatnot. I copy those virgin GPU client folders into Dagny's C:\FAH\MACHINES\NEWNODENAME\ folder. Still with me? Now I have C:\FAH\MACHINES\NEWNODENAME\GPU0 through C:\FAH\MACHINES\NEWNODENAME\GPU7 sitting there ready to fire up a client for the first time, make the WORK directory, download the fah_core, download work, etc.

In Dagny's C:\FAH folder (the "root" of the F: share) I have a simple batch file called RUN8.BAT (I also have RUN1, RUN2, RUN4, RUN6, etc.):

Code:
START CMD /C "F:\GPU\GPURUN.BAT 0"
START CMD /C "F:\GPU\GPURUN.BAT 1"
START CMD /C "F:\GPU\GPURUN.BAT 2"
START CMD /C "F:\GPU\GPURUN.BAT 3"
START CMD /C "F:\GPU\GPURUN.BAT 4"
START CMD /C "F:\GPU\GPURUN.BAT 5"
START CMD /C "F:\GPU\GPURUN.BAT 6"
START CMD /C "F:\GPU\GPURUN.BAT 7"


Now, in the GPU folder under the share I have the GPU F@H client renamed "GPU.EXE" and its DLLs along with a simple batch file called GPURUN.BAT:

Code:
@ECHO OFF
TITLE %USERDOMAIN% - GPU %1
@ECHO Changing to directory F:\MACHINES\%USERDOMAIN%\GPU%1
F:
CD F:\MACHINES\%USERDOMAIN%\GPU%1
@ECHO COPYING NEW DLL FILES (IF ANY)
XCOPY /D /Y F:\GPU\*.DLL F:\MACHINES\%USERDOMAIN%\GPU%1
@ECHO STARTING GPU%1 IN %CD%
F:\GPU\GPU.EXE -gpu %1 -verbosity 9 -advmethods -forceasm -forcegpu nvidia_g80 %2 %3 %4 %5 %6


See, that batch file uses the %USERDOMAIN% command shell macro to move to the correct MACHINE directory on the network share and then the 0-7 command line variable from the first batch file to switch to the correct GPU# directory on the network share.

F:\MACHINES\%USERDOMAIN%\GPU%1
  becomes something like
F:\MACHINES\REARDEN\GPU3

I create a shortcut to the F:\RUN8.BAT batch file on the desktop of the new client. Now when I click on the shortcut it runs the batch file with all the START commands in it which starts 8 shells by calling the second batch file 8 times. The second batch file changes to the correct directory for that specific GPU client, specifies all of the command line parameters for the client and adds a nifty title to the CMD window so I can easily tell the clients apart at a glance (i.e. "REARDEN - GPU 3")

Notice though that there are extra batch file command line macros %2-%6 after the standard command line. These additional command line options will be empty (null) if the RUN8.BAT batch file doesn't use them and therefore have no effect, but if you want to have some GPUs run with additional options you can copy the RUN8.BAT file instead of creating a shortcut to it. Then in the copy of the batch file on your desktop you can add additional command line options to customize the behavior of some GPU clients to be different than the default. This gives a bit more flexibility to the system -- not every client must run the same command line.

Since every single GPU client ultimately uses GPURUN.BAT to start, I can change every client's command line in one central location. Since every GPU client uses the same GPU.EXE to run, I can update the client software by replacing one executable file.

I arrived at this solution after much gnashing of teeth trying to keep the 34 clients in the rack coordinated. Originally I had discrete setups on each node's drive, but I was making many errors trying to manually keep them synched.

Here's the neat part. When I add a new node the only thing I have to do to the local machine is map \\DAGNY\FAH to F:, copy a shortcut to F:\RUN8.BAT to the desktop and double click it. That's IT. There is no F@H software on the client machines unless you want to run SMP.

In summary the entire node addition procedure is:
1. Go to node Dagny, C:\FAH\MACHINES
2. mkdir NEWNODENAME
3. Copy GPU0-GPU7 virgin dirs from USB stick to C:\FAH\MACHINES\NEWNODENAME
4. Go to new machine, map \\DAGNY\FAH to F:
5. Create shortcut to F:\RUN8.BAT on new client's desktop
6. Click shortcut and watch 8 console GPU clients spring up

Finished product looks like this (click to enlarge):

Image

Jason


Last edited by Atlas Folder on Fri Jul 16, 2010 2:33 pm, edited 14 times in total.

Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 2:42 pm 
Monkey Federation (Top 10)
Monkey Federation (Top 10)
User avatar

Joined: Tue Oct 11, 2005 5:20 pm
Posts: 10472
Location: Kansass
Wow again.

That was way too much for my little head to digest.

Do the machines have to be on a domain, or can they be on a home network 'workgroup'?


Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 2:56 pm 
8086
8086

Joined: Sat Mar 14, 2009 10:02 pm
Posts: 14
Mine are all in WORKGROUP. Doesn't matter as long as you can map the share.

Jason


Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 3:38 pm 
Folding for Elena
Folding for Elena
User avatar

Joined: Sat Jan 29, 2005 11:18 pm
Posts: 8537
Location: Lookin' at you from outside the cage
if i may be sincerely direct here...

THAT'S JUST FUCKING AMAZING... GREAT JOB...

you not only answered my question but about 20 others that haven't even occurred to me yet.... thanks for the info.... it's fun watching how people work with the f@home clients.... if i may ask as well what do you do for a living...

very impressive mang...


Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 7:19 pm 
8086
8086

Joined: Sat Mar 14, 2009 10:02 pm
Posts: 14
staypuff,

Necessity is the mother of invention. It was all fun and games until I added nodes 3 and 4. Then just logging in to each client and double clicking 8 start icons was getting ridiculous. I had to come up with a solution. I still have to log in to each machine to start the 8 clients since the batch file must be executed in the context of the machine running the clients, but now it's just logging in, clicking one icon and logging out.

About OS's, the best OS for GPU folding on GTX295s or 9800GX2s is Windows XP 32-bit or 64-bit. Vista CUDA is bugged at detecting the second GPU in these dual GPU cards (unless it's been fixed very recently). Some folks have intermittently gotten Vista to work with dual GPU cards but they're the minority.

Another compelling benefit of XP is that you do not need dummy plugs to fake a monitor on each of the cards' outputs as you do with Vista. In Vista you cannot extend the desktop onto a given output unless a monitor is detected. This is a Microsoft WHQL requirement, and while nVidia could program around it, it would mean that their driver could not receive WHQL certification. I've seen some people writing about a method around needing dummy plugs in Vista but I honestly haven't paid much attention since the issue doesn't affect me any longer.

Jason


Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 8:50 pm 
Folding for Elena
Folding for Elena
User avatar

Joined: Sat Jan 29, 2005 11:18 pm
Posts: 8537
Location: Lookin' at you from outside the cage
thanks Jason... i've been with XP on everything so far and it's been extremely easy to setup on this os.... my "soon to be here" 295 will be on XP.... i've noticed this on the Stanford Folding forums that most of the people that were claiming success were using XP....

Excellent job by yourself to streamline the process... you are to be congratulated.... going to extremes in projects like this is honorable...

you sure you aren't a monkey????

we have room in our clubhouse??? :wink:


Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 11:04 pm 
Folding for Elena
Folding for Elena
User avatar

Joined: Sat Jan 29, 2005 11:18 pm
Posts: 8537
Location: Lookin' at you from outside the cage
oh just one other thing.... any idea why the flag -g80 has to be added at the end???? i've seen some that say you don't have to add it and others that say you must... considering this isn't remotely a g80 gpu any idea why this flag is needed at all???? really curious as there seems to be differences in opinion out there in folding land... i'm going to add this flag myself as you've had obviously good success... :wink:


Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 11:16 pm 
Million Club - 5 Plus*
Million Club - 5 Plus*
User avatar

Joined: Sun Sep 12, 2004 6:37 pm
Posts: 4745
Location: In the monkey's litterbox
staypuff wrote:
oh just one other thing.... any idea why the flag -g80 has to be added at the end???? i've seen some that say you don't have to add it and others that say you must... considering this isn't remotely a g80 gpu any idea why this flag is needed at all???? really curious as there seems to be differences in opinion out there in folding land... i'm going to add this flag myself as you've had obviously good success... :wink:


You shouldn't need it on Windows. It's more there for the Linux/GPU folders so that the GPU detection always passes.


Top
  Profile  
 
 Post subject:
PostPosted: Tue Mar 17, 2009 11:36 pm 
Folding for Elena
Folding for Elena
User avatar

Joined: Sat Jan 29, 2005 11:18 pm
Posts: 8537
Location: Lookin' at you from outside the cage
smartcat99s wrote:
staypuff wrote:
oh just one other thing.... any idea why the flag -g80 has to be added at the end???? i've seen some that say you don't have to add it and others that say you must... considering this isn't remotely a g80 gpu any idea why this flag is needed at all???? really curious as there seems to be differences in opinion out there in folding land... i'm going to add this flag myself as you've had obviously good success... :wink:


You shouldn't need it on Windows. It's more there for the Linux/GPU folders so that the GPU detection always passes.



ok.... i'm just curious as to why everyone is using these flags.... i'm also curious as to why Stanford hasn't said a fucking thing about it one way or the other... you'd think they'd try to come up with a proper guide..... it's like a gaming company asking you to pay for the game then then telling you to code it yourself..... i've seen 3 different ways to add flags for dual gpus and some work and some don't..... :roll:

i'll try both ways when the card comes in... you'd think that people that donate time and money to a project wouldn't have to figure it out for themselves however.... :wink:


Top
  Profile  
 
 Post subject:
PostPosted: Wed Mar 18, 2009 5:32 am 
8086
8086

Joined: Sat Mar 14, 2009 10:02 pm
Posts: 14
I used to never use "-forcegpu nvidia_g80". However, when building one of my machines I spent a couple of hours once trying to figure out why the GPU client kept telling me that my GPU wasn't supported. It was behaving as though SLI was still engaged even though it definitely wasn't. Eventually I tried this flag even though I'd never needed it before and it fixed the problem. So now I just include it all the time.

Jason


Last edited by Atlas Folder on Wed Mar 18, 2009 11:18 am, edited 1 time in total.

Top
  Profile  
 
 Post subject:
PostPosted: Wed Mar 18, 2009 10:20 am 
Team Member Top 250
Team Member Top 250
User avatar

Joined: Tue Apr 15, 2008 8:01 pm
Posts: 1260
Dude

We are not worthy dude


Top
  Profile  
 
 Post subject:
PostPosted: Wed Mar 18, 2009 3:32 pm 
Team Member Top 500
Team Member Top 500
User avatar

Joined: Mon Oct 03, 2005 11:56 am
Posts: 969
Location: The Koprulu Sector
nekollx wrote:
Dude

We are not worthy dude


heh my x1900xtx burned up rendering that pic in jealousy


Top
  Profile  
 
 Post subject:
PostPosted: Wed Mar 18, 2009 7:29 pm 
Team Member Top 50
Team Member Top 50
User avatar

Joined: Fri May 16, 2008 1:05 am
Posts: 2437
Location: Sold my soul to Satchboy for an avatar
smartcat99s wrote:
staypuff wrote:
oh just one other thing.... any idea why the flag -g80 has to be added at the end???? i've seen some that say you don't have to add it and others that say you must... considering this isn't remotely a g80 gpu any idea why this flag is needed at all???? really curious as there seems to be differences in opinion out there in folding land... i'm going to add this flag myself as you've had obviously good success... :wink:


You shouldn't need it on Windows. It's more there for the Linux/GPU folders so that the GPU detection always passes.


I've seen people who've had trouble with their clients - after looking at their log someone points out that the client thinks they have ATI card. They add the flag and problem goes away.
I didn't need it on my first multi GPU rig. But I use it all the time now. Even though your card isn't really a g80, the g80 is the only recognized parameter for the "-forcegpu" flag by the client. I've seen people SAY they use g92, but g92 doesn't actually exist to the client. There are only the nvidia_g80, then two different ATI processors.

Don't know if that helps clear up some of the other recommendations you've seen puffy?

staypuff wrote:
... you'd think that people that donate time and money to a project wouldn't have to figure it out for themselves however.... :wink:
So, now you consider yourself a people? Eh, monkey?


Top
  Profile  
 
 Post subject:
PostPosted: Thu Mar 19, 2009 5:13 pm 
8086
8086

Joined: Thu Mar 19, 2009 5:04 pm
Posts: 1
Atlas Folder wrote:
Ah, PSU's... Well in the first shelves that I built I used a single 1,250W Cooler Master power supply. But the shelf draws ~1,050W under full load and there are 4 12V rails on that PSU so I am really pushing them pretty hard.

The 1,250s are holding up fine, they don't complain at all, but I am a bit worried about them under such a high load plus they got "deactivated" over at newegg. So I switched to dual 750W PC Power & Cooling supplies that each have a single 12V rail. I use a cable that allows me to switch the two supplies on simultaneously. Working great so far with one powering the motherboard and two GTX295s while the other powers the hard disk and two GTX295s.

Jason

Jason,

Great work. Could you share more specifics about the 2x750W and the cable?

I know Jaak (Estonia Donates) had difficulties with PSUs also.

Thanks,

Denny


Top
  Profile  
 
 Post subject:
PostPosted: Thu Mar 19, 2009 6:05 pm 
Folding for Elena
Folding for Elena
User avatar

Joined: Mon Jul 14, 2008 8:41 am
Posts: 458
Location: Folding as Lyle_Goodwin
Now...not to seem obtuse for making the suggestion...but, you have your mission critical drive in a RAID 1, no? Or a RAID 5?

Seems to be definitely the place for such a thing...especially since we're talking about a very high concentration of heat, and relatively high IOs...

Actually, seems like a good place to put a solid state disk, now that I think about it, but, again, don't want to seem to mock your hardware, or your purpose for doing it...

It all seems so...noble. In an awesomely nerdy way.


Top
  Profile  
 
 Post subject:
PostPosted: Thu Mar 19, 2009 7:29 pm 
8086
8086

Joined: Sat Mar 14, 2009 10:02 pm
Posts: 14
Hey guys,

@CPUacreage:
In my first two shelves I used a single 1250W Cooler Master PSU. This power supply can accommodate 4 cards with 4 4-pin and 4 8-pin PCIe power connectors. I have not had a single problem running those entire machines on those supplies, but since the power draw of the system is so high (~1140W with SMP running) and those supplies do not have a single 12V rail there is no question that I'm pressing them fairly hard. Also, they got "deactivated" on newegg.com. That was alright though, because I'd been planning on moving to a dual supply solution anyhow.

Now I'm using a pair of PC Power & Cooling 750 80+ certified SLI PSUs. I wire those two supplies to switch together with Lian Li Dual Power Supply Adapter, plug 2 cards into one supply and two cards into the other and it just works. I also put the HDD on one and the motherboard on the other to even out the load a bit further. This has worked perfectly, I can't tell I'm using two supplies. I've now built two shelves this way.

Jaak and I have talked about this stuff a little bit in emails and on the F@H forums.

@Ilander
If the conversation is civil and constructive I don't mind at all talking about this stuff. You're neither mocking or obtuse; don't sweat it.

This stuff isn't exactly Mission Crititcal. It won't affect my ability to make a living or injure someone if it fails. Nevertheless, setting up a task to copy that primary disk's FAH directory tree off to one of the secondary machines at midnight or something isn't a bad idea.

There's nothing on the primary disk other than an OS, a few easily downloadable F@H files and the work units. If that disk died I would cuss it, hurl it into the trash, slap a new one in and reinstall Windows XP x64. And although folks talk about heat all the time when they see my system, it really isn't that hot, especially with the new 5900 series WUs which run much cooler.

I did consider solid state disks when I first started the project, especially when I saw their prices had dropped to under $100.00. But then I saw that the cheapest spinning drives were $34.00 for 80GB; can't beat that with a stick. I'm paying that much for the RAM, it's really amazing. I thought about trying to make the remote nodes diskless and do network booting, but at $34.00 a node for the disks it just not worth the hassle.

Hope that clarifies some stuff for you guys.

Jason


Top
  Profile  
 
 Post subject:
PostPosted: Thu Mar 19, 2009 7:44 pm 
Team Member Top 50
Team Member Top 50
User avatar

Joined: Fri May 16, 2008 1:05 am
Posts: 2437
Location: Sold my soul to Satchboy for an avatar
Ilander wrote:
....Actually, seems like a good place to put a solid state disk, now that I think about it, .....
Actually, it wouldn't be a good place for SSD. SSD's have a significantly lower life expectancy that spinning platter HD's - in terms of the number of times a location on the drive can be written over. With so many GPU work folders turning clients around so quickly, you would kill that drive (or drives) in no time.

My $.02 on the matter.


Top
  Profile  
 
 Post subject:
PostPosted: Thu Mar 19, 2009 7:52 pm 
Team Member Top 50
Team Member Top 50
User avatar

Joined: Thu Nov 16, 2006 1:46 am
Posts: 831
Location: California
Everything is all fine and dandy but.........for the most MaximumPC fun you need to run the 11108 "flag". :wink:


Top
  Profile  
 
Display posts from previous:  Sort by  
Post new topic This topic is locked, you cannot edit posts or make further replies.  [ 63 posts ]  Go to page Previous  1, 2, 3, 4  Next

All times are UTC - 8 hours


Who is online

Users browsing this forum: No registered users and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group

© 2014 Future US, Inc. All rights reserved.