Swap file on Ramdisk?

[Yamyam]

Right, right. It doesn't make much sense to be splitting up my pastures and sending the doggies and cowpokes back and forth between them..... but the alternative is to send 'em all the way down the Chisolm Trail to Kansas City and back, which makes even less sense.

So, back in the real world of RAM allocation, I can see that Windows demands a swapfile whether it needs it or not.
And since I've got plenty of RAM, I want to trick Windows into using a dummy swapfile that points right back into another partition of RAM. I'm no OS guru, but it seems like this ought to give a pretty good improvement in overall system performance. Is that a reasonable conclusion, or am I still missing something here? If the swapfile is being used (ie, larger than zero size), then by definition it's got data that the system needs to access from time to time.


Ok, so here's an interesting idea for a mobo designer..... what if you add another level of cache, which would live between the system RAM and the hard drive? For instance, add an onboard hardware RAID controller with several slots for cheap PC100/133 SDRAM cache, and configure your BIOS so that one or more of the slots can be recognized as a RAMdisk for the swapfile before Windows boots. You're looking for performance, not reliability, so you can use standard SDRAM without any of the expensive features like ECC, registers, or battery backup. In fact, you don't even need the RAID functionality, but you may as well include it as long as you've got a processor for the cache already. I'd guess that you could implement it for an added retail cost of $50 per board, which isn't unreasonable at all in a world where 8mb-cache IDE drives and software-RAID mobo's can bring a $20-30 premium.

 

[Dug]

No, it's up to the OS, the application never specifies to use the swap file, it just turns out some API's require them.

So if the programmer decides to use these API's isn't that the same as what I just said? Or should I refrase that- the programmer has decided to use the swapfile. not the program.
BTW if you run NWN it will tell you that you swap file is too small before you even play the game if you've manually set it to some number. And guess what? it crashes if that number is too low. Doesn't matter how much ram you have. So don't tell me a program can't specifically use a swap file on purpose.
If all memory is dished out by the OS then tell me how much does each app get?
Photoshop seems to be able to gobble it up. Not to mention you can set up your own swap file for it.

And seperating your memory for that swap file would seem impossible without it being on a seperate bus.

Exactely what I said.
Or should I say a different channel, bank, memory controller.

 

[Oogle]

Correct me if I'm wrong here, but if you decrease the prime land by reserving it for a RAMdisk, don't things get tight a whole lot sooner? Doesn't that mean the memory manager has a smaller realm of good lots? He's gonna get mad about that and have to do a lot more work putting perfectly good pages out to pasture. (The fact that this particular pasture happens to be a kludgey 'fast' pasture is lost on him.) You can't directly mount (access) pages out in pasture, however fast that is, without first moving them back into real RAM. That sounds like a whole lot of work shuffling things between parts of your land. You'd have to hire lots more cowboys and dogs and fences and the like. If it's all your land to start with, why fence some of it off for the explicit purpose of making things harder on yourself?

I don't know why the NT memory manager thinks some pages smell so bad that they need to be stored outside when the inside is so sparsely populated. Barring rewriting the manager-- remember linux this is not, shouldn't we just accept that the manager might have a good reason to throw a few pages outside too soon? Can't we assume that they're pretty useless pages to begin with?

But the problem is that Windows is not using that "prime land" at all. Just realize that memory access is 1,000,000 times faster than disk access. The idea of using a "swapfile on RAMDisk" should solve 2 problems:
1. It forces Windows to finally use RAM that I paid good money for.
2. It resolves page faults 1,000,000 times faster in RAM than on disk.

What's wrong with those ideas?

BTW if you run NWN it will tell you that you swap file is too small before you even play the game if you've manually set it to some number. And guess what? it crashes if that number is too low. Doesn't matter how much ram you have. So don't tell me a program can't specifically use a swap file on purpose.

So why would a program have to use the disk as RAM on purpose? I just don't see the rationale at all behind the performance that is gained by using disk instead of RAM.

 

[Nothinman]

That doesn't work so well with XP

My experiences with XP (no pun intended) are limited, it was only on my notebook for maybe 3 weeks.

In fact there is no perceivable overhead whatsover. (relative to speed of machine) Even if there was some that was noticable, it would be a lot faster than having to readjust it manually and reboot.

The speed of the hard drive is the main factor, and on my laptop the speed of the disk sucks. I've always had a big enough pagefile, so it's not even an issue.

t's much easier to use the swap file, which is limitless (as long as its dynamically controlled). That way they don't have to rewrite code for every configuration out there (which would be very hard to do.) And even if they did, it might not be as efficient as you may think.

That's a stupid assumption. Disk space is not limitless. The program should never care where the memory came from, just malloc() or new or whatever and let the OS handle it from there.

Let's say you did have a program that used up your main memory and didn't use a swap file. What would happen if you loaded up IE, or Word, or something else? How much memory is that program going to have to give up and send to the swap file so another program can run? It doesn't know. It would be a giant guessing game. And in the end it would yield slower results, more tech calls, etc.

The OS pages to disk when it needs to, the program has no idea whether all of it's memory is in memory or on disk and it doesn't need to. When it requests memory that's been paged to disk a pagefault happens and the OS figures out where to load the data from, it then restarts the program a long and the program has no idea what happened.

If I was writing a program for the general public, I sure wouldn't want to do that.

No offense, but if you really believe that, you're an idiot. Modern OS design contradicts that 100%.

As I recall, programmers in Windows can set a tag in their code that says that certain bits of code can be swapped out to disk and some shouldn't. This is such that the programmer can say what is important and should stay in memory until it is really needed for something else, and what can be dumped

That may be true, but even with those 'hints' you may be paged to disk if memory is needed, you can't stop that from happening.

I don't think even Administrator accounts can mlock() pages by default in NT.

Ack... now I'm really confused about this issue. First I hear that devs explicitly control the memory to get the best performance.

That's because Dug is severely confused about how modern OSes work. bsobel and I are correct, the app may say "please don't swap me out" but in the end the OS has the final say and can page anything to disk it wants to free up physical memory.

The memory manager will try to use RAM as much as possible and not swap out UNLESS
1. The subsystem wants to use the disk as memory to perform its operations.
2. Apps tell the OS that it wants to use the disk as memory.

1. If memory is needed and there is none, something will be paged out. The OS decides what gets paged to disk, not the applications. pagefile space is not directly usable as memory so the OS has to be carefull what it pages out.
2. Apps don't ask for pagefile space, they get paged out as the OS decides necessary.

But, how can a system be "stressed" if several memory-hogging apps combined take only 1/4 of the total system memory(1 GB)?

My only guess is that NT tries to be proactive and swap somethings out early "just in case". Like I said, a good NT internals book will probably explain everything if you really want to learn.

 

[Smilin]

Dude check it out. You're not really taking anyone's word for it here are you? You've asked a question and been given an answer. If you want more details then feel free to take an operating systems design class (I took mine at Purdue..it's not bad feel free to look it up).
There are a LOT of variables in how an operating system manages memory. It's quite frankly the most important thing that an operating sytem actually does. There is also a lot more going on than just memory allocation for programs (did you know each share on your PC takes up some mem?). Say for instance you have a ram disk like you want and you're running a bunch of apps together. One app loads something from disk... the os drops the data in memory and also reads the next group of clusters off the disk in an attempt to be proactive about your needs. Ooops! just ran out of memory! What to do? well I'll just drop these pages over to the pagefile (which should go really quickly because we're using a ramdisk right?) It does...fast as snot, but then I'm still in the 'low memory' situation that caused me to page in the first place and the next app is bitching for cpu cycles...ok, shuffle those pages off the ram disk and into 'normal' memory. Now back to the first app...which is low on memory and needs paged a bit. Next thing you know you're thrashing (yes, a technical term). Sure your ramdisk is fast but because your 'main' memory is low your cpu is spending more time moving pages back and forth to it's percieved virtual disk than actually processing apps. Keep in mind my friend that NT allocates 4 GIGS to EACH app that is running regardless of how much physical memory and disk space exist.
The solution you propose would make paging faster for sure but it would also increase the need for paging which may have not been necessary without. 'Tuning' an OS isn't trivial and is far beyond any skills I'll ever have. Don't second guess the designers...just buy a bunch of ram and let the OS handle it.

This isn't windows 95 you're running, it's Windows NT 5.10

 

[Dug]

So how can some programs use tons of physical ram and very little swap file usage (or none) and some programs use very little physical ram and a large swap file?

If the program isn't dictating it and the os is, why is the os favoring some apps and not others?
If the os has complete control woudn't every app get the same amount of memory?

 

[Oogle]

My only guess is that NT tries to be proactive and swap somethings out early "just in case". Like I said, a good NT internals book will probably explain everything if you really want to learn.

Now that I think about it, I'm not really looking for an in-depth technical explanation on how the NT memory manager works (which I'm assuming a techy NT book would give me). I'm just looking for the general philosophy behind its implementation and strong evidence that backs that philosophy. NT trying to be proactive "just is case" sounds like a pretty good guess, but a more concrete and "official" response with scenarios to support it would be greatly appreciated.

One app loads something from disk... the os drops the data in memory and also reads the next group of clusters off the disk in an attempt to be proactive about your needs. Ooops! just ran out of memory! What to do? well I'll just drop these pages over to the pagefile (which should go really quickly because we're using a ramdisk right?) It does...fast as snot, but then I'm still in the 'low memory' situation that caused me to page in the first place and the next app is bitching for cpu cycles...

But now you're talking about an environment where my system would be low on memory. I'm not talking about that. I'm talking about an OS that decides to swap memory out to disk just for the hell of it even though I have 700 MB of RAM available. How is having 700MB RAM free considered by the memory manager to be "low on memory"? Using a RAMDisk right now is just a hacky, band-aid way of telling the OS, "Hey, wake up! I have plenty of RAM right here. Stop f#$%ing around with my disk and use the RAM I spent my hard-earned cash on."

Don't get me wrong about virtual memory. I love the idea of virtual memory, but only when my system really needs it. Does a *nix OS follow the philosophy of using as much RAM as possible before having to resort to swapfile access?

 

[Nothinman]

If the program isn't dictating it and the os is, why is the os favoring some apps and not others?

Maybe because MS codes in favor of their apps? I don't know, you'd have to ask MS or perhaps the NT Internals book I keep mentioning would help. I've 99.9% Linux now so that's what I know inside and out (and having the source helps a lot), NT's vm subsystem implementation is different but the basic ideas are still the same.

If the os has complete control woudn't every app get the same amount of memory?

No, because notepad with no file open needs a lot less memory than Word with a 120page document open does. The OS allocates memory as needed.

Does a *nix OS follow the philosophy of using as much RAM as possible before having to resort to swapfile access?

Even Linux will use some swap proactively, to save time for when it does have to page to the swap partitions. This is called cached swap, something that exists both in swap space and in memory, it can be easily discarded from memory in the event it's needed instead of wasting a lot of cycles looking for good things to free at the last minute.

But I have 1.2G memory in my machine at home, and both NT and Linux rarely page anything to the swap space.

 

[bsobel]

Originally posted by: Dug
So how can some programs use tons of physical ram and very little swap file usage (or none) and some programs use very little physical ram and a large swap file? If the program isn't dictating it and the os is, why is the os favoring some apps and not others?
If the os has complete control woudn't every app get the same amount of memory?

I've already answered this, there are certain api's (such as unnamed memory backed files) that the OS implements in terms of the swap file. That doesn't mean (per say) the programmer said 'use the swap file', but rather the design they picked used the api functionality, the OS implementation takes care of the details (in this case using swap area for the backing store)

Bill

 

[Oogle]

Even Linux will use some swap proactively, to save time for when it does have to page to the swap partitions. This is called cached swap, something that exists both in swap space and in memory, it can be easily discarded from memory in the event it's needed instead of wasting a lot of cycles looking for good things to free at the last minute.

But I have 1.2G memory in my machine at home, and both NT and Linux rarely page anything to the swap space.

So what happens in the "here and now"? You say that on your system(s), NT and Linux rarely pages anything. I understand that to mean that most/all of your processes each use around 0 to 1 MB of virtual memory (i.e. swap space). So am I just misunderstanding Task Manager when it says that my process is using 32MB of memory (mem usage) and 32MB of swap space (vm size). Is the process really mostly all in RAM instead of on disk? I'm curious to know what you see when you use Task Manager to view your processes' memory usage. For a typical memory hogging process, do you see 64MB mem usage, ~0 vm size?

 

[Flatline]

Back to the original question, which was whether using a ramdisk for swap should increase performance...theoretically, it could. One of the quickest ways to get a performance boost out of Windows 2000 (not as familiar with XP) is to put your pagefile on the system's fastest physical disk and, if possible, on a non-system partition (because the system partition gets accessed more than others). If your system sees the ramdisk as just another hard drive, then you should be able to place your pagefile on that ramdisk and make use of the faster medium. With 1GB of memory, he should be able to allocate a large portion of it for swap space and still have more than enough to run the system on. The only problem is that it would be possible to run out of space on that ramdisk (that's a lot of memory usage, but it COULD happen), and when that happens Windows gets very unhappy very quickly; this is, I assume, why the more knowledgeable persons in this thread are advising against it. Personally, I agree with the cautious types simply because I prefer stability to performance. For the most part, any "preemptive" swapping that NT memory managers do is for background processes and doesn't really effect system performance that much.

Got to admit, though, that I am finding this thread quite informative and interesting.

 

[bsobel]

Originally posted by: Flatline
Back to the original question, which was whether using a ramdisk for swap should increase performance...theoretically

Again, it's better to use that memory as system file cache and as general memory than allocate it as a ramdisk.
Bill

 

[sechs]

Originally posted by: Oogle
So am I just misunderstanding Task Manager when it says that my process is using 32MB of memory (mem usage) and 32MB of swap space (vm size). Is the process really mostly all in RAM instead of on disk? I'm curious to know what you see when you use Task Manager to view your processes' memory usage. For a typical memory hogging process, do you see 64MB mem usage, ~0 vm size?

In Windows NT, all memory is virtual. Applications don't see any real memory, just the great little space which has been created for them.

I have yet to divine any relationship between memory usage and virtual memory size.

 

[Flatline]

I agree...for stability's sake if for no other reason; all I said was that it SHOULD be possible to slightly increase (and it would be slight, because as I said, with that much memory primarily background processes are swapped anyway) performance by doing that.

I never advocate performance tweaks that could sacrifice stability, and this one theoretically could.

 

[Oogle]

Again, it's better to use that memory as system file cache and as general memory than allocate it as a ramdisk.

But what if you had a lot of memory? For example:
- System file cache has reached a stable level (~120 MB)
- Several of my favorite apps are running (300 MB)

I have 1GB or RAM total. I would still have 600 MB of RAM left over. The OS memory manager decides "Screw all that available RAM. I want to use disk because insert MS explanation here!". Wouldn't part of that 600MB leftover of wasted free RAM be better used for a swapfile RAMDisk? In other words, wouldn't it be time for me to say, "Stop that. Here's some RAM. Use this, not the disk"?

The only way I see this example not happening is when:
- The system file cache will never stop growing and will eventuallyhurt performance if it isn't allowed to get larger than 300MB.
- I decide to run 50 more (useless) additional apps on my system.

Seriously, with 1GB of RAM, isn't it safe to say that there is plenty of room for the sys file cache, process memory, and a "RAMlike" swapfile? If not 1GB, how about 2GB? I'm sure there's no hard and fast rule that says that it's always better to use that memory as system file cache and as general memory than allocate it as a ramdisk. Right?

 

[Nothinman]

I have yet to divine any relationship between memory usage and virtual memory size.

I'd be curious to see what they mean by 'vm size' too. My only guess is that maybe 'mem usage' is the apps RSS and that 'vm size' is the total memory the application has set aside, including memory requested but not allocated and mmap'd files (other dlls, exes, etc too).

The OS memory manager decides "Screw all that available RAM. I want to use disk because insert MS explanation here!". Wouldn't part of that 600MB leftover of wasted free RAM be better used for a swapfile RAMDisk? In other words, wouldn't it be time for me to say, "Stop that. Here's some RAM. Use this, not the disk"?

All you'll probably end up doing is moving the point where the NT VM decides to swap to a lower point, now instead of 400M it decides to do it at 200M. You're no better off and you're probably worse off because you're creating unnecessary memory contention.

I'm sure there's no hard and fast rule that says that it's always better to use that memory as system file cache and as general memory than allocate it as a ramdisk. Right?

Technically there is, because if you're paging to disk the OS needs the memory for something else so decreasing the available memory would be the opposite of what you want to do.

 

[Oogle]

I'm sure there's no hard and fast rule that says that it's always better to use that memory as system file cache and as general memory than allocate it as a ramdisk. Right?

Technically there is, because if you're paging to disk the OS needs the memory for something else so decreasing the available memory would be the opposite of what you want to do.

LOL. This is beginning to sound like a "chicken before the egg" scenario. You're saying that OS needs the available memory for something. So I say, "Fine. Don't use the RAMDisk." I'll give the OS all the free RAM it wants so it can do that "something else". Now, it decides not to use it. You've got to be kidding me.

The OS memory manager decides "Screw all that available RAM. I want to use disk because insert MS explanation here!". Wouldn't part of that 600MB leftover of wasted free RAM be better used for a swapfile RAMDisk? In other words, wouldn't it be time for me to say, "Stop that. Here's some RAM. Use this, not the disk"?

All you'll probably end up doing is moving the point where the NT VM decides to swap to a lower point, now instead of 400M it decides to do it at 200M. You're no better off and you're probably worse off because you're creating unnecessary memory contention.

Does it really matter that it swaps out to RAMDisk? Sure, you'll run into problems with additional page faults. But that's an easy sacrifice to make when the page fault resolution now occurs 1,000,000 times faster in memory than on disk. I don't think page faulting wastes the same magnitude of time as disk access when you're resolving your faults in the memory domain.

 

[Nothinman]

LOL. This is beginning to sound like a "chicken before the egg" scenario.

Kind of. The main problem would be that NT initializes it's pagefiles before all the drivers (I think) so it would be very difficult to place a pagefile on a ramdisk since the ramdisk wouldn't be setup in time.

But really it's a VM tuning issue. It seems like MS tuned NT to run with 256-512M memory, and as such it swaps things out 'just in case' even when you'd think it shouldn't. Maybe if NT's VM was more user tunable or we had the source, we'd know for sure but neither are true.

Sure, you'll run into problems with additional page faults.

Page faults aren't problems.

If you think you can make it work, give it a try. Most of us think it's really stupid even if you could get the pagefile to work on the ramdrive, but maybe we're crazy. Although this whole thing seems like a big band-aid for a poorly tuned VM system.

 

[opk]

I believe that most all of us agree that the RAMdrive thing is a silly idea. Those of us that have had a little training are trying to explain why this is to those who haven't.

Page Faults aren't a bad thing. Everyone has been saying the right things. NT allocates 4 GB of "space" (really addresses) for each process. Very few actually use it, but thanks to 32 bit addressing, it's there. Individual processes don't know where any of they're addresses physically reside, nor do they care. If they happen to be waiting for a while they just get booted from the processor, and when their turn comes around again, it might be there. Remember that all this happens VERY FAST. (news to some of you) The processor runs MANY processes constantly, even when we don't want it to. The OS is MADE to swap out processes. That' is its job, it is good at it. Page faults happen ALL THE TIME. You NEVER notice them if you have lots of RAM. You might notice a higher priority process (usually a system process) hogging all your cycles, but that has nothing to do with Page Faults. PFs were a big thing in Win9X, but only because it was a baby operating system made for small computers. The NT kernel is a good, modern kernel. It can handle this job. Luckily, we all can run modern OSs and NOT HAVE TO WORRY about silly little things like page faults.

 

[Oogle]

Originally posted by: opk
I believe that most all of us agree that the RAMdrive thing is a silly idea. Those of us that have had a little training are trying to explain why this is to those who haven't.

Hopefully, this summarizes the "against-ramdisk" explanations I've heard so far..
- Using a RAMDisk as a swapfile takes away memory that your OS would otherwise use.

The problem I have with that explanation is this: If I give NT that memory back, it won't use it. If NT is not going to take advantage of it, then I'm going to take advantage of it. How? By force feeding the RAM back to NT via a RAMDisk. Why is that a problem? NT isn't using it. It will most likely never use it unless I throw 200 apps and services at it, which I know I'll never do. I need some help understanding this better. All I know is that fundamentally, RAM is faster than disk. Anything I do to access off of RAM instead of the disk is a good thing for performance. How is this fundamental concept no longer true?

 

[opk]

Fundamentally RAM is faster.

In reality given enough memory the NT memory manager accomplishes all its jobs well. It keeps the system from crashing. It protects memory between processes. It keeps the system from bogging until absolutely necessary. It causes no discernalbe slowdowns in its normal operation. It workswell and that is that.

Back to Mr. Yam's original point. No, it's not possible to put the swap file in RAMdisk. Further, it's a silly idea.

How much would this improve the (subjective) performance of my PC?

None. In fact, it would decrease performance.

 

[Oogle]

Originally posted by: opk
Back to Mr. Yam's original point. No, it's not possible to put the swap file in RAMdisk. Further, it's a silly idea.

I haven't actually tried to even make a RAMdisk on NT/XP, let alone put a swapfile on it. But, it sounds like you have tried and failed in doing so. How did you go about trying to do this? Also, could you or someone please explain why it's "silly" or "stupid"? All I'm reading right now are quick retorts, but no clear explanations to support them...

How much would this improve the (subjective) performance of my PC?

None. In fact, it would decrease performance.

Could you please explain how? Doesn't the fact that it will decrease performance contradict the fact that using RAM is faster than disk?

 

[bsobel]

All I'm reading right now are quick retorts, but no clear explanations to support them...

Was: Putting it nicely, someone recommended you go take an OS internals class. Go do that, then we'll all be happy to discuss this in painfull detail with you.

Ok, I'm home now and not as annoyed this thread is still going on Someone (I think it was you) asked about the vm size column in device manager, I think there is a mistaken belief that the vm size column indicates how much memory is being stored in the swap file. It does NOT indicate this. In fact, in your 1gig example, your probably not paging at all (except paging in code as needed). This is unless your using an application (Photoshop comes to mind) that uses unnamed memory mapped files. Photoshop does this so it can successfully allocate a large CONTIGOUS block of virtual memory, even on low physical memory systems. It's the correct behaviour for what it needs to do, but actually not common for most applications.

Note that not current using the swap file doesn't mean the system hasn't created a swap file so you could use it if needed. For example, this machine is a dual 2.8 xeon with 2 gigs of memory, but I also have a 2 gig swap file. So, I could allocate upto 4 gigs at one time without running out of memory. However, my current load is 359 meg used (the other 1.5 or so gig is system cache right now). None of that 359mb is hitting the swap file, I wouldn't expect the swap file to start getting used until I hit closer to 1.5 gig used (maybe a bit more, maybe a bit less).

Bill

 

[Oogle]

Putting it nicely, someone recommended you go take an OS internals class. Go do that, then we'll all be happy to discuss this in painfull detail with you.

Frankly, all the arguments against a ramdisk so far haven't gone over a topic that would warrant me to "take an OS internals class". The arguments I keep hearing are...

- It's silly or stupid
- Using a RAMDisk as a swapfile takes away memory that your OS would otherwise use.

The first argument is more of a comment. The 2nd argument doesn't make any sense because the OS would never use any of that available memory. It appears to swap out processes for no reason. To alleviate the disk penalties of this unnecessary swapping, you would try to swap to a faster medium, like a RAMDisk. It's a band-aid fix, yes. But, I still don't see what's wrong with doing it. Judging from the answers so far, it appears that no one else here really knows why it's wrong either. I hope that's not true.

 

[Lorne]

It seems its not impressing him, So just turn off the Swapfile and save the ramspace and dont use the ramdrive.

Its all pointless to try.