Swap file on Ramdisk?

[opk]

Here's the point Oog. Yes, RAM is faster than Hard disk space. On the order of 1,000,000 times so. The point is that the geniuses that wrote the NT memory manager (or all modern ones) already know that. Since the NT memory manager is written to utilize RAM first, and only page out to disk when necessary, there is no provision for (nor point to) a RAMdisk.

Assuming you had a BIOS addressable solid state RAMdisk you are more than welcome to mount it as your R drive (or whatever) and throw your swapfile on that. Have fun running your benchmarks to find out that the whole endeavour is pointless. Whatever you spent on the RAMdisk would have been better spent on more primary system memory.

RAMdisks were hot stuff back when compuers weren't 32-bit (to allow 4GB of RAM); HDs were small, slow and expensive; and OSs weren't 'modern' enough to have real memory managers. There is a reason you don't hear about them anymore.

peace-out

 

[Oogle]

Ok, I'm home now and not as annoyed this thread is still going on Someone (I think it was you) asked about the vm size column in device manager, I think there is a mistaken belief that the vm size column indicates how much memory is being stored in the swap file. It does NOT indicate this. In fact, in your 1gig example, your probably not paging at all (except paging in code as needed). This is unless your using an application (Photoshop comes to mind) that uses unnamed memory mapped files. Photoshop does this so it can successfully allocate a large CONTIGOUS block of virtual memory, even on low physical memory systems. It's the correct behaviour for what it needs to do, but actually not common for most applications.

Note that not current using the swap file doesn't mean the system hasn't created a swap file so you could use it if needed. For example, this machine is a dual 2.8 xeon with 2 gigs of memory, but I also have a 2 gig swap file. So, I could allocate upto 4 gigs at one time without running out of memory. However, my current load is 359 meg used (the other 1.5 or so gig is system cache right now). None of that 359mb is hitting the swap file, I wouldn't expect the swap file to start getting used until I hit closer to 1.5 gig used (maybe a bit more, maybe a bit less).

Ok, great. Thanks for the clarification on VM Size. But, now I have many more questions...

- If VM Size isn't the right value to look at, what is? How can I see how much of a process' memory is swapped out to disk? Wintop for 9x used to do this pretty well. I was just wondering if Task Manager or Performance Monitor had the same facilities.

- If a Photoshop-like app runs on a 1+GB system, would an allocated contiguous block of virtual memory map straight into the RAM instead of RAM and disk?

- If little/none of your swap file is being used, why did you make a 2 gig swap file?

- Why would a game like NWN (or any app in general) force a user to allocate 300 MB of swapfile? Does that mean that the game really is trying to use 300 MB of swap space and the OS is causing the app to fail? Or is it more of an artificial limit set by developers?

- Can I safely ignore any low VM warnings if I'm using a fixed size swapfile? If not, why not?

- What does NT consider to have more priority, System File Cache or Available Memory? If more memory was needed, would NT reduce the SysFile Cache size to accommodate or would it start using the swapfile? Is there a way to tune the priority?

The point behind using a RAMdisk was to just make sure that Windows is using an overwhelming majority of its memory in RAM, not disk. But if Windows is already doing that like you say it is, then the whole point is moot. I only wish now that could have been said earlier.

 

[bsobel]

The 2nd argument doesn't make any sense because the OS would never use any of that available memory. It appears to swap out processes for no reason.

Yes it would.

Geez,
Bill

 

[Nothinman]

It seems its not impressing him, So just turn off the Swapfile and save the ramspace and dont use the ramdrive.

Having no pagefile is only slightly better. There is no reason not to have a pagefile.

- If little/none of your swap file is being used, why did you make a 2 gig swap file?

He works for a company that write some windows drivers (or whatever they call the filesytem hooks) if he does debugging on his box I'm sure he'd like a full memory dump for examination.
And otherwise maybe he just needs the ability to have ~4G of memory allocated at one time.

Disk space is cheap, I had ~1.5G pagefile and I have 1.2G memory, I had 4x400M pagefiles IIRC. And I have SCSI drives and I think disk space is cheap =)

- Can I safely ignore any low VM warnings if I'm using a fixed size swapfile? If not, why not?

If the OS is bitching, it wants more and if you don't give it to it it'll eventually fail on something.

- What does NT consider to have more priority, System File Cache or Available Memory? If more memory was needed, would NT reduce the SysFile Cache size to accommodate or would it start using the swapfile? Is there a way to tune the priority?

Any memory available is available to be used as filesystem cache, as soon as a process needs it it'll be flushed and reallocated, not paged to disk. The point of a filesystem cache is to causes less disk I/O, not more.

The point behind using a RAMdisk was to just make sure that Windows is using an overwhelming majority of its memory in RAM, not disk. But if Windows is already doing that like you say it is, then the whole point is moot

That's what we're trying to say. Until you understand VM tuning better than the OS developers at MS, you have to trust them.

 

[Barnaby W. Füi]

and the moral of the story is, leave the damn thing alone already. you can only tweak your five billahertz rocketship so much. is there really that big of a difference between Fast and Fast+0.000001 anyways? come on.

 

[bsobel]

- If VM Size isn't the right value to look at, what is? How can I see how much of a process' memory is swapped out to disk? Wintop for 9x used to do this pretty well. I was just wondering if Task Manager or Performance Monitor had the same facilities.

You really can't without some real digging, I think perfmon might be able to give some clues, but it's not tracked in the way your thinking of it.

- If a Photoshop-like app runs on a 1+GB system, would an allocated contiguous block of virtual memory map straight into the RAM instead of RAM and disk?

The issue isn't the machine having a gig of memory free, the issue is photoshop needing enough contigous free memory in it's address space (a whole different discussion). Unnamed memory mapped files are one way of doing this, and is what some developers have used.

- If little/none of your swap file is being used, why did you make a 2 gig swap file?

I didn't, I let the OS do what it wanted to do. My boot volume is dual striped WD 120's, I currently (after system restore, hibernation, paging, etc) still have about 188gig free. So, I'm not really sweating letting the OS do what it wants. And as Nothinman pointed out, I sometimes need full memory dumps, so I need the swap file big enough to accomidate that.

- Why would a game like NWN (or any app in general) force a user to allocate 300 MB of swapfile? Does that mean that the game really is trying to use 300 MB of swap space and the OS is causing the app to fail? Or is it more of an artificial limit set by developers?

I've not seen that specific game, but I have seen programs check for a minimum because they determine that the program doesn't gracefully handle low memory situations. It's kinda blanket safety net for them. It doesn't mean the app is neccesarily using that space.

- Can I safely ignore any low VM warnings if I'm using a fixed size swapfile? If not, why not?

If the OS complains that your low on memory, then your low on memory. You can ignore it, the results when you actually 'run out' of memory should be gracefull application recover, but in practice 'bad things occur'.

- What does NT consider to have more priority, System File Cache or Available Memory? If more memory was needed, would NT reduce the SysFile Cache size to accommodate or would it start using the swapfile? Is there a way to tune the priority?

The file system cache shrinks to some minimum size giving priority to application memory. You can adjust this, but you shouldn't need to (even tho you might see that 1/2 your memory (or 3/4 in my current case) is allocated to caching, the system will take that right down real quick if it needs the memory. The memory manager will (too a very high degree) trade of disk cache to avoid incurring paging overhead).

The point behind using a RAMdisk was to just make sure that Windows is using an overwhelming majority of its memory in RAM, not disk. But if Windows is already doing that like you say it is, then the whole point is moot. I only wish now that could have been said earlier.

Sorry, re-reading the thread it could have been clearer...

Bill

 

[sechs]

So, have we got this? This is a silly idea because it doesn't speed up your system; it slows it down!

 

[Gantry]

Edit --> Wrote a nice response that seemed to clarify where Oogle & everyone else was, and replied to it before I noticed there was a page two! Time for sleep...

Fun thread, will re-read tomorrow...

 

[bsobel]

yet the swap file is being heavily accessed.

That wasn't established. I think the confusion came from the assumption that the virutal size column in device manager indicated the amount of swap file storage used. As I earlier pointed out, it does not indicate this.

Bill

 

[Gantry]

Yeah bsobel, was trying to get both sides on the same page (no pun intended again), unfortunately I wrote that reply without noticing all the replies on page2. But you saw it before my edit, my apologies...

And I agree with you - I don't think there was as much swap being used as he perceived...

 

[Dug]

No, it's up to the OS, the application never specifies to use the swap file, it just turns out some API's require them.
So the program doesn't ask for these API's?

The program should never care where the memory came from
It seems like a lot of games, photo, and cad programs do though. Note I said 'seems'.

The OS pages to disk when it needs to, the program has no idea whether all of it's memory is in memory or on disk and it doesn't need to
I understand the os pages when it needs to, but Photoshop comes to mind on the second part.

The OS decides what gets paged to disk, not the applications
Apps don't ask for pagefile space, they get paged out as the OS decides necessary.
NWN does and and I believe Baldur's Gate II. If you manually set your file to 200MB, even if you have a gig of ram the game will crash. Which contradicts 'the application never specifies to use the swap file' and 'the program should never care where the memory came from'. I remember back in the day you could manually change the swap file usage by the original Baldur's Gate.

there are certain api's (such as unnamed memory backed files) that the OS implements in terms of the swap file. That doesn't mean (per say) the programmer said 'use the swap file', but rather the design they picked used the api functionality, the OS implementation takes care of the details (in this case using swap area for the backing store)
No matter the details it still sounds like the programmer has the ability to use the swap file when he or she want to.

This is unless your using an application (Photoshop comes to mind) that uses unnamed memory mapped files. Photoshop does this so it can successfully allocate a large CONTIGOUS block of virtual memory, even on low physical memory systems. It's the correct behaviour for what it needs to do, but actually not common for most applications.
Sounds like an app is programmed to use the swap file to me.

The issue isn't the machine having a gig of memory free, the issue is photoshop needing enough contigous free memory in it's address space (a whole different discussion). Unnamed memory mapped files are one way of doing this, and is what some developers have used.

That's because Dug is severely confused about how modern OSes work.

Gee I wonder why? There seems to be some contradictions here. I'm obviously not a programmer so I only report what I see. If I see a program use a large swap file, but I still have 800MB of physical ram free, that 'looks' like the app is controlling it, not the OS. Why? becuase if the OS was controlling it then I would assume it would use up more physical ram and make the swap file smaller. (Just observation)

 

[Dug]

Oh and here's your swap file made out of ram-

Platypus Technology
Clikc on products then qikcache.

"In applications that use disk space for caching and virtual memory, the average performance gain of almost 800% was obtained over a conventional drive."
PC World, September 2000.

"Using 2GB with QikCACHE can increase effective CPU usage up to 84% compared to the measly 3.3% usage that you would get with using a 10,000 rpm SCSI 9GB HDD."
CAD User, March 2001

"Using VM on the HDD achieved 432 pages per second through the VM swap file. After QikCACHE was installed, this skyrocketed to 9,126 pages per second!"
CAD User, March 2001

"Using VM on the HDD achieved 432 pages per second through the VM swap file. After QikCACHE was installed, this skyrocketed to 9,126 pages per second!"
CAD User, March 2001

Not as fast as system ram, but still a notch up from hd's.

 

[bsobel]

No matter the details it still sounds like the programmer has the ability to use the swap file when he or she want to.

No, it means the developer has used one of a number of valid techniques on the win32 platform to achieve a certain goal. The OS he is running on my implement that in terms of the system swap file, but it doesn't have to. This is typically used when the process needs to share data with another process (such as a background render implemented as a seperate process in some drawing programs).

Oh and here's your swap file made out of ram-

Solid state harddrives are not what we are discussing here, and a wastefull solution based on system memory vs solid state drive costs.

Bill

 

[Dug]

Solid state harddrives are not what we are discussing here, and a wastefull solution based on system memory vs solid state drive costs

No Sh!t. I was just offering a link to a ram drive.
Which is not too far from what was discussed here. In fact it's exactely what I said earlier in the post, that you would need a swap file on a different memory controller, bus, channel, or bank, etc for it to work.

and in your words -
means the developer has used one of a number of valid techniques on the win32 platform to achieve a certain goal

Well if the certain goal is using the swap file, and a programmer uses valid techniques, that sure sounds a lot like the programmer can use the swap file when he or she wants.

 

[Smilin]

Hey Dug.

Go put a ramdisk on your system, move the pagefile there and tell us how it runs.

I'm beginning to get the feeling you'de rather play devils advocate than actually listen to any answer that anyone gives.

Go do it now before this thread reaches 3 pages.

 

[Dug]

Like I said, I was just offering a link. No reason to get pissy cause its not the best technology.

 

[Nothinman]

No Sh!t. I was just offering a link to a ram drive.
Which is not too far from what was discussed here. In fact it's exactely what I said earlier in the post, that you would need a swap file on a different memory controller, bus, channel, or bank, etc for it to work.

I think his point was that while it is a ramdisk and you could probably put a pagefile on it, it would be stupid because of the cost/performance ratio. Since you have so much memory already you gain virtually nothing by putting a pagefile on that thing. It would be much more worthwhile to put something else on there that you need fast access to, like say digital video or even the OS if you can boot from it.

means the developer has used one of a number of valid techniques on the win32 platform to achieve a certain goal

Well if the certain goal is using the swap file, and a programmer uses valid techniques, that sure sounds a lot like the programmer can use the swap file when he or she wants.

What he's saying is that the developer can use certain system calls to ensure that he gets X amount of contiguous virtual memory (virtual addresses as opposed to real addresses, not pagefile space). If the OS decides it needs to page something out to make that allocation happen, that's an OS implementation detail and can vary from OS to OS, hell MS may make the next revsion of NT do something different and the app won't care at all.

Take unnamed memory mapped files for instance. Normally the OS uses the file on disk as the backing store, so that when memory is needed it can just discard the memory mapped file and reuse the memory because it knows it can just reload it from disk when it needs to. With unnamed memory mapped files it can't just discard the memory because there's no disk file to reload from, so it puts the data in the pagefile so it has the ability to free that up when necessary. It may never need to do that and may just discard the pagefile data when the app is closed, but then again it may not.

Note: I'm not an NT VM expert, anything I say should be taken with a grain of salt and verified with another source before being labled as actually true =)

 

[Dug]

I'm beginning to get the feeling you'de rather play devils advocate than actually listen to any answer that anyone gives.

You are correct, I would rather play devils advocate, but I do listen to the answers.

What he's saying is that the developer can use certain system calls to ensure that he gets X amount of contiguous virtual memory (virtual addresses as opposed to real addresses, not pagefile space). If the OS decides it needs to page something out to make that allocation happen, that's an OS implementation detail and can vary from OS to OS, hell MS may make the next revsion of NT do something different and the app won't care at all.

Take unnamed memory mapped files for instance. Normally the OS uses the file on disk as the backing store, so that when memory is needed it can just discard the memory mapped file and reuse the memory because it knows it can just reload it from disk when it needs to. With unnamed memory mapped files it can't just discard the memory because there's no disk file to reload from, so it puts the data in the pagefile so it has the ability to free that up when necessary. It may never need to do that and may just discard the pagefile data when the app is closed, but then again it may not.

Thanks, that makes sense.

 

[bsobel]

No Sh!t. I was just offering a link to a ram drive.

Doug, no need to get snippy. You didn't offer a link to a ram drive, you offerend a link to a solid state storage device. COMPLETELY different from the topic of the whole thread.

Bill

 

[Dug]

Doug, no need to get snippy. You didn't offer a link to a ram drive, you offerend a link to a solid state storage device. COMPLETELY different from the topic of the whole thread.

Ok no need to get picky about my wording. Its a drive of ram which is in other words a solid state storage device. Or a virtual memory swap space made out of ram, or sdram based virtual memory. (companies description)

And because the idea of moving your swap file from the hd to your system memory hasn't been done (at least no one here has pointed to the ability to do it) I pointed to an alternative which is specifically made to offer a swap file for your OS made out of ram. Which I believe isn't that far off topic.

 

[bsobel]

Originally posted by: Dug

Doug, no need to get snippy. You didn't offer a link to a ram drive, you offerend a link to a solid state storage device. COMPLETELY different from the topic of the whole thread.

Ok no need to get picky about my wording. Its a drive of ram which is in other words a solid state storage device. Or a virtual memory swap space made out of ram, or sdram based virtual memory. (companies description)

And because the idea of moving your swap file from the hd to your system memory hasn't been done (at least no one here has pointed to the ability to do it) I pointed to an alternative which is specifically made to offer a swap file for your OS made out of ram. Which I believe isn't that far off topic.

Doug, I'm not being picky about your wording. These are two COMPLETELY different things. And this 'alternative' is NOT specifically made to offer a swap file for your OS made of RAM. It's made to provide very fast disk I/O for extremely specific applications. The cost/benefit comparision is a joke for that device compared to adding memory.

Bill

 

[Dug]

From their website

Efficiently add incredibly fast swap space and .tmp file storage to enhance your system's speed and efficiency.

QikCACHE can be configured to function exactly like the virtual memory in servers and desktop PCs - but without the mechanical limitations of the system's hard disk drive. In fact, the QikCACHE range delivers unprecedented performance in this area.

A system's virtual memory is nothing more than allocated HDD space, that operating system and software applications are swapped in and out of during use. Both the traditional size of this swap space, and the limiting speed of the system's hard drive can create data processing 'choke-points'.


High-performance:
Takes advantage of the PCI bus - for sustained data throughput speeds of up to 110MB/sec.
Scalable:
QikCACHE2 offers between 512MB and 2GB. QikCACHE8 offers between 1GB and 8GB. Additional QikCACHE products can be added and spanned to create even large storage space for future growth and system capacity needs.
Set and forget:
Driver software automatically creates and formats QikCACHE at system start-up, creating a permanent swap-file configuration.
Flexible:
Two slots in QikCACHE2 and eight slots in QikCACHE8 accept Platypus SDRAM DIMMs of 512MB and 1GB.

Workgroup servers:
Boost workgroup server performance, smooth peak loads, get faster response from operating system and applications.
Engineers/CAD designers:
Increases the productivity of demanding CAD programs while maintaining current hardware.
Desktop/workstation power users:
Overcomes swap file bottlenecks, increases high speed cache, allows more software programs to run simultaneously, enables faster graphic redraws.
High data security workplaces:
Maintains a secure workstation or terminal, with no magnetic traces of data when machine is powered down.

Departmental file & application servers:
Obtain maximum value from server investments via substantially increased system performance.
Web/proxy servers:
Web hosting, caching, proxy servers, reverse proxy servers, streaming data serving.
Terminal Server environments:
Swap file space for application servers.
Database servers:
Maximise performance of critical database applications.


So doesn't this replace your swap file? It's just a card that fits in your pci slot with slots on it to fill with sdram. (plus other chips obviously)
Couldn't tell ya how much it is though because I can't find one for sale.

 

[Nothinman]

So doesn't this replace your swap file? It's just a card that fits in your pci slot with slots on it to fill with sdram. (plus other chips obviously)
Couldn't tell ya how much it is though because I can't find one for sale.

According to them it can, but really if swap is your bottleneck this isn't the right solution. And I'm sure if you found one for sale the price would be too prohibitive to even consider buying one.

It would be better used on a disk that has a lot activity like a busy mail spool, news spool, scratch space for DV editing, etc.

 

[sechs]

Originally posted by: Dug
A system's virtual memory is nothing more than allocated HDD space

Based on this statement alone, I would not believe anything that the company says about that product. Obviously, they know nothing about how operating systems work (especially Windows), and are just selling to idiots with big ballances.

 

[opk]

Dug,

You happen to be right about this. BUT, it's for extreme situations, not for the original question. This product sounds like a good idea IF your mobo memory is maxed out and you still need more. This obviously doesn't happen much as most mobos accept 1.5-2 GB of main memory. Most people (in excess of 99% i'd hazard) don't need that much.

So, assuming you're sitting there with this monster set of applications that you just have to run and your machine is maxxed out at 2GB of RAM. The applications really want, say, 3 GB of RAM. In that specific case, this product might be worth the money. It isn't the same as main system RAM, it isn't near as efficient, but it is many many times faster than paging out to hard disk.

Big servers and huge databases can need large amounts of RAM. That is behind the push (really more of a nudge) toward 64-bit computing. It's main benefit right now is it breaks the 4GB addressable RAM limit. But, again, we're talking ebay big here, nothing you, oog, or yam need worry their pretty little heads about.

This is a different topic than the original one yam broached. He wondered about excess main system memory. This product is explicitly for use when main system memory is at capacity.