Superfetch + Readyboost explained. *LONG*

[bsobel]

Originally posted by: Jeff7181
Here's an example of what I'm talking about. See the Disk Defragmenter creating disk activity? See the Search Indexer creating disk activity?

When Superfetch is creating disk activity the "Image" will be svchost.exe and the "File" will be what ever file it's caching.

BTW, in that screen shot that isn't superfetch acessing the prefetch folder, its the cache manager...

 

[Jeff7181]

Originally posted by: Smilin

Originally posted by: Jeff7181
Here's an example of what I'm talking about. See the Disk Defragmenter creating disk activity? See the Search Indexer creating disk activity?

When Superfetch is creating disk activity the "Image" will be svchost.exe and the "File" will be what ever file it's caching.

Jeff I don't follow this.

There are several instances of svchost and each holds several services. It would be more complicated than that to figure out what hosted service is doing what. You can take a good guess depending on the file I guess.

That screenshot doesn't even show svchost though. It's got searchindexer highlighted which should be indexing your disk to speed up search results. I'm not sure this is at all related to superfetch. Also, the disk activity is like 60,000-70,000 Bytes per min. A modern drive will do 60,000,000 Bytes per *second* so whatever is going on it isn't much. Your defragger appers to be chugging away there. It's background though so if you decide to do something that will fall quickly.

Again, I'm just a bit confused.

That particular screenshot was for n7 because he couldn't find what I was looking at. I created disk activity with Disk Defragmenter to show n7 what it looks like.

I didn't capture an image of the problem I'm describing to the rest of you because it slowed the computer to a crawl and taking a screen shot probably would have taken 15 minutes.

And you're right... I took a good guess based on what I knew to be true. The indexer has it's own process name which you can see in the screen shot I provided for n7, disk defragmenter has it's own process name which you can also see in the screen shot provided, and WoW has it's own process name. The only other thing I could think of that would access the disk so heavily was Superfetch... and it made sense that it would be doing that since I had been playing WoW for a while which accesses that file quite often, so it probably figured it would be a good idea to cache it.

The problem with that is that the file won't fit in RAM even if all my RAM is free... what good is to to cache part of a file? More importantly, HOW would it cache only part of a file and why would it try to cache a file that's too big to fit into RAM? As far as Windows is concerned, that 3.56 GB file is a single file.

bsobel made a good point that Superfetch is a background process and should stop whatever it's doing when another program needs resources. That didn't seem to be the case because the computer slowed to a crawl and the hard disk light was on, solid. When I finally got the Reliability/Performance monitor to load so I could see what was thrashing the disk so badly, it was svchost.exe accessing the file C:\Program Files\World of Warcraft\Data\Common.MPQ... it had read over 1.2 GB total at the time I looked at it, and the number was rising. No matter what I did it wouldn't stop, so I rebooted, which took a LONG time.

After rebooting and getting back to the desktop, I noticed a lot of disk activity again, so I look, and the same damn thing is happening. svchost.exe reading Common.MPQ... 300 MB read... 350... 400... etc.

So I did some quick thinking, opened the properties for the C:\Program Files\World of Warcraft\Data folder and disabled indexing for the folder and all subfolders and files. The disk activity stopped immediately and all returned to normal.

One might conclude that the Indexing service was responsible, since I disabled it and the problem went away. But if one looks a little deeper (as I did) and sees that the Indexing service has it's own process name, and Disk Defragmenter has it's own process name, Superfetch is the only other thing I can think of that would access a file like that. So... maybe that Indexing checkbox in the properties window also controls what Superfetch will or won't cache. Ever since I cleared that checkbox, I've been able to run WoW without that issue popping up again.

 

[bsobel]

More importantly, HOW would it cache only part of a file and why would it try to cache a file that's too big to fit into RAM? As far as Windows is concerned, that 3.56 GB file is a single file.

See here is the problem. Your making false assumptions about how the OS works. The statement 'as far as windows is concerned that 3.56gig file is a single file' is completely false. The cache manager (and prefetch/superfetch) work at the page level, not the file level. The system only prefetches the parts of a file that are actually used. As an example, if you run Winword and never execute certain code paths, the code backing those code paths will NOT be prefetched.

One might conclude that the Indexing service was responsible, since I disabled it and the problem went away. But if one looks a little deeper (as I did) and sees that the Indexing service has it's own process name, and Disk Defragmenter has it's own process name, Superfetch is the only other thing I can think of that would access a file like that. So... maybe that Indexing checkbox in the properties window also controls what Superfetch will or won't cache. Ever since I cleared that checkbox, I've been able to run WoW without that issue popping up again.

Wow, just wow. I'll simply say that the index property has no effect on Superfetch. You seemed destined to blame it regardless of the lack of proof and the obviousness that it's the indexer (as I already suggested). Worse, you think you understand system internals to a point that you can debug this, but you don't and you continue to make logic jumps (such as the above) with have no basis in fact.

Worse, your terribly hard to help as you add new information everytime you post. That new information completely changes the issue your asking for help on.

 

[bsobel]

One more thought...

Lastly, the indexer is supposed to throttle back IO as well, strange that is isn't. Do you have any other sw on the box that might be indexing (Google search for example) that might also be respecting the index setting?

 

[Jeff7181]

your terribly hard to help as you add new information everytime you post.

I'm not asking for help. I solved the problem I had.

 

[rseiler]

After reading the last several posts, I was reminded of yet another reason I knew it wasn't the indexer doing the relentless file reading on my system (I gave the other reasons above): the files in question are not even on a drive within the scope of the indexer.

I'd be happy to use Superfetch again when MS offers a way (even if it's via the Registry) of telling it what/where to ignore. Until then, it's like someone's in the background always running file copies, and it's not productive.

 

[bsobel]

Originally posted by: rseiler
After reading the last several posts, I was reminded of yet another reason I knew it wasn't the indexer doing the relentless file reading on my system (I gave the other reasons above): the files in question are not even on a drive within the scope of the indexer.

I'd be happy to use Superfetch again when MS offers a way (even if it's via the Registry) of telling it what/where to ignore. Until then, it's like someone's in the background always running file copies, and it's not productive.

It seems strage that your the only person who seems to be having this issue. Jeff's is clearly tied to the indexer (regardless of the I turned off indexing and the problem went away, therefore I have proven it's the prefetcher logic). What is the contents of your prefetch folder, those are the files the system will be prefetching.

Dont get me wrong, I believe you when you say something is wrong on your system, I just think you've identified the wrong thing...

 

[rseiler]

Note that I never said there was anything wrong with my system, but rather that there's something wrong -- or less than intelligent -- about Superfetch.

The prefetch folder has nothing in it newer than 5 days ago, when I turned off Superfetch.

I couldn't possibly have identified the wrong thing for the reasons I've previously mentioned. It's very clear what the problem is, and I don't understand how there can even be a discussion about it at this point. The indexer finished its business after about a day, and it's clearly identified by the task name. You know when it's running because of that. The disk has been quiet since. If I turn on Superfetch and start accessing files, the disk reading of *those same files* will commence shortly thereafter. If I disable it, it stops. It can't possibly get clearer than that.

As was pointed out to me in another thread (or maybe here, I don't recall), those with 1GB or less will not be as susceptible to this issue because there's very little (if any) RAM left over for prefetching. On systems with 2GB or more, Superfetch can go crazy, and it does.

 

[bsobel]

On systems with 2GB or more, Superfetch can go crazy, and it does.

Thats the thing, I have 32gig so I should see the world being paged in by your theory and I'm not....

 

[HardWarrior]

Michael Fortin on Superfetch.

 

[lxskllr]

Originally posted by: HardWarrior
Michael Fortin on Superfetch.

Thanks :thumbsup: Good discussion there.

 

[bsobel]

Originally posted by: rseiler
Note that I never said there was anything wrong with my system, but rather that there's something wrong -- or less than intelligent -- about Superfetch.

I didn't mean to suggest something is wrong with your hardware or anything like that. But if you look at the thread you have two people who claim to have SF problems, one is clearly the indexer. The other (yours) sounds like a real issue. However, you've decided that since your having a problem the implementation is stupid and must not work for anyone with more than 2gig of memory. However it clearly is working fine for the vast majority of users (my self included) that have more than 2gig of memory on their box. As such, my comment about 'wrong with your system' was in regards the the way SF is working, as that is definately not normal and I am still wondering why.

 

[HardWarrior]

Originally posted by: lxskllr

Originally posted by: HardWarrior
Michael Fortin on Superfetch.

Thanks :thumbsup: Good discussion there.

You're welcome! I just wish the interviewer would have let Fortin babble more.

 

[Smilin]

Originally posted by: Jeff7181

Originally posted by: Smilin

Originally posted by: Jeff7181
Here's an example of what I'm talking about. See the Disk Defragmenter creating disk activity? See the Search Indexer creating disk activity?

When Superfetch is creating disk activity the "Image" will be svchost.exe and the "File" will be what ever file it's caching.

Jeff I don't follow this.

There are several instances of svchost and each holds several services. It would be more complicated than that to figure out what hosted service is doing what. You can take a good guess depending on the file I guess.

That screenshot doesn't even show svchost though. It's got searchindexer highlighted which should be indexing your disk to speed up search results. I'm not sure this is at all related to superfetch. Also, the disk activity is like 60,000-70,000 Bytes per min. A modern drive will do 60,000,000 Bytes per *second* so whatever is going on it isn't much. Your defragger appers to be chugging away there. It's background though so if you decide to do something that will fall quickly.

Again, I'm just a bit confused.

That particular screenshot was for n7 because he couldn't find what I was looking at. I created disk activity with Disk Defragmenter to show n7 what it looks like.

I didn't capture an image of the problem I'm describing to the rest of you because it slowed the computer to a crawl and taking a screen shot probably would have taken 15 minutes.

And you're right... I took a good guess based on what I knew to be true. The indexer has it's own process name which you can see in the screen shot I provided for n7, disk defragmenter has it's own process name which you can also see in the screen shot provided, and WoW has it's own process name. The only other thing I could think of that would access the disk so heavily was Superfetch... and it made sense that it would be doing that since I had been playing WoW for a while which accesses that file quite often, so it probably figured it would be a good idea to cache it.

Hm, couple of bad guesses and assumptions there could have led you astray early in your troubleshooting.

Here you've assumed that Windows would try to cache more than will fit into ram. Quite an outlandish assumption...

The problem with that is that the file won't fit in RAM even if all my RAM is free... what good is to to cache part of a file? More importantly, HOW would it cache only part of a file and why would it try to cache a file that's too big to fit into RAM? As far as Windows is concerned, that 3.56 GB file is a single file.

Actually as far as Windows is concerned that file is ~900,000 4k clusters. That may or may not matter here.

bsobel made a good point that Superfetch is a background process and should stop whatever it's doing when another program needs resources. That didn't seem to be the case because the computer slowed to a crawl and the hard disk light was on, solid. When I finally got the Reliability/Performance monitor to load so I could see what was thrashing the disk so badly, it was svchost.exe accessing the file C:\Program Files\World of Warcraft\Data\Common.MPQ... it had read over 1.2 GB total at the time I looked at it, and the number was rising. No matter what I did it wouldn't stop, so I rebooted, which took a LONG time.

It is indeed a background process. Which svchost were you looking at? How did you tell which hosted service in that svchost was the one reading common.mpq? You know that things like the workstation service share the same svchost as superfetch right? If you did identify which svchost it was, are you sure that blizzards patcher wasn't simply reading that file with the workstation service? Again, lots of assumptions with no data.

After rebooting and getting back to the desktop, I noticed a lot of disk activity again, so I look, and the same damn thing is happening. svchost.exe reading Common.MPQ... 300 MB read... 350... 400... etc.

Are you running scheduled backups by chance? Did you know they'll try to catch a missed event the next time you boot? Not saying this is what was happening, just trying to point out that there are more possibilities than superfetch as the culprit. Without evidence you don't really know.

So I did some quick thinking, opened the properties for the C:\Program Files\World of Warcraft\Data folder and disabled indexing for the folder and all subfolders and files. The disk activity stopped immediately and all returned to normal.

One might conclude that the Indexing service was responsible, since I disabled it and the problem went away. But if one looks a little deeper (as I did) and sees that the Indexing service has it's own process name, and Disk Defragmenter has it's own process name, Superfetch is the only other thing I can think of that would access a file like that. So... maybe that Indexing checkbox in the properties window also controls what Superfetch will or won't cache. Ever since I cleared that checkbox, I've been able to run WoW without that issue popping up again.

You appear to really have latched onto superfetch at this point. In order to support your theory you are now saying that "maybe" something completely unrelated is now somehow related. If you do A and B happens it's common to assume A causes B. May or may not be right but it's not a bad place to start. Here you are doing A, seeing B and concluding C.

I can think of a lot of things that would hit your disk like that. If Superfetch is the only thing you can think of I can understand why you would have assumed. You gotta know when you are assuming and when you aren't though or you'll head down the wrong path a lot.

I'm not trying to say you're wrong here. You could be right. Based on the info provided it's not possible to tell. What you said to bsobel earlier did irk me a bit:

"The problem is that you want to be right so badly that you won't acknowledge the problem, which is being experienced by multiple people. /shrug"

I'm a professional troubleshooter and based on what you've said in this thread you are in no position to really tell what the problem is. Taking the next step and bashing someone (especially someone like bsobel) based on a bunch of assumptions is a bit much.

I'm glad you're working now at least. Bummer that you may have unnecessarily disabled some indexing though.

 

[rseiler]

Originally posted by: bsobel

Originally posted by: rseiler
Note that I never said there was anything wrong with my system, but rather that there's something wrong -- or less than intelligent -- about Superfetch.

I didn't mean to suggest something is wrong with your hardware or anything like that. But if you look at the thread you have two people who claim to have SF problems, one is clearly the indexer. The other (yours) sounds like a real issue. However, you've decided that since your having a problem the implementation is stupid and must not work for anyone with more than 2gig of memory. However it clearly is working fine for the vast majority of users (my self included) that have more than 2gig of memory on their box. As such, my comment about 'wrong with your system' was in regards the the way SF is working, as that is definately not normal and I am still wondering why.

I think it comes down to a difference of opinion on the pain level involved in this. I'm one who doesn't appreciate this kind of disk activity, while others apparently don't mind it, or see it as a good return on the investment.

If I was just accessing small stuff all day, I probably *wouldn't* mind it, since the disk reading wouldn't be as noticeable. But as I've mentioned a couple times, some of the files I access, and which SF dutifully tries to pre-cache at a later time, are very large and are just not appropriate/worthwhile for SF.

MS is aware that some want user configurability on this. Maybe it's not that hard to implement and we'll see it in a future version.

 

[bsobel]

I think it comes down to a difference of opinion on the pain level involved in this.

I think your missing my point, something appears to be wrong on your system as other users aren't seeing the same thing. What it is thats causing this to occur, I don't know, but am curious to figure out.

But as I've mentioned a couple times, some of the files I access, and which SF dutifully tries to pre-cache at a later time, are very large and are just not appropriate/worthwhile for SF.

Trust me, I probably deal with much larger files than use on a regular basis (4-200gig vms) and simply am not seeing this. Based on your symptoms I should be, that has be a bit confused...

 

[delco007]

Thank you , for giving me this information . I needed to know this from a long time . I wasnt critical about the way Microsoft have built the new Operating system . Its becoz most of the reviews that i have come across accuse vista of one thing or another .

bye and take care
Guru

Originally posted by: Smilin

Originally posted by: delco007
Hi Mark,
I loved the way you have explained the working of primary and secondary memory but it reminded me of a question which has been ringing at the back of my head for a looong time .
I have heard that these features were already present in a certain MAC os , if that is correct then wouldnt it be true that Microsoft has copied their features ? Also i would like to know if Microsoft can be credited for making a good Operating system or its just that they have copied features from here and there and Jumbled them together and named it their new Operating system ???

I would love to know the truth .

Bye for now
Guru .

If it was true wouldn't that mean..??? Go find out if it is true first. :roll:

I would not be surprised if apple is working on something similar to ready booste and superfetch. If they have it already I would be interested to know if it is superior or inferior.

If Apple has such a thing I would be very surprised if they came up with the idea first. Apple is spending like $534 million in research (2005 numbers I found). Last year Microsoft spent $6.5 billion (with a B) and filed 3,467 patents.

Heck Apple didn't even manage to fully design the iPod. Microsoft owns several patents used in the iPod interface.

As far as MS copying features from others I'm sure some of that occurs in both directions. However Vista was already in development when the very first OSX was released back in 2001. OSX was of course not developed from the ground up by Apple either. Apples OS's and hardware were falling further and further behind Wintel so they finally discarded the OS (and backwards compatibility as they've done several times) and designed a new OS based on FreeBSD (others work). To remedy the hardware situation they are now switching to "windows hardware" in the form of the x86.

If we should start accusing companies of copying others why don't we start by accusing Apple of stealing computer designs from Fisher Price and Playschool.



..Good Post by the way BD!

 

[BD2003]

Originally posted by: rseiler

Originally posted by: bsobel

Originally posted by: rseiler
Note that I never said there was anything wrong with my system, but rather that there's something wrong -- or less than intelligent -- about Superfetch.

I didn't mean to suggest something is wrong with your hardware or anything like that. But if you look at the thread you have two people who claim to have SF problems, one is clearly the indexer. The other (yours) sounds like a real issue. However, you've decided that since your having a problem the implementation is stupid and must not work for anyone with more than 2gig of memory. However it clearly is working fine for the vast majority of users (my self included) that have more than 2gig of memory on their box. As such, my comment about 'wrong with your system' was in regards the the way SF is working, as that is definately not normal and I am still wondering why.

I think it comes down to a difference of opinion on the pain level involved in this. I'm one who doesn't appreciate this kind of disk activity, while others apparently don't mind it, or see it as a good return on the investment.

If I was just accessing small stuff all day, I probably *wouldn't* mind it, since the disk reading wouldn't be as noticeable. But as I've mentioned a couple times, some of the files I access, and which SF dutifully tries to pre-cache at a later time, are very large and are just not appropriate/worthwhile for SF.

MS is aware that some want user configurability on this. Maybe it's not that hard to implement and we'll see it in a future version.

Superfetch is intelligent in a "trys it's best" sense, but it's one size fits all.

Yes, its SF thats accessing that huge file. Why? Because you must have used it often. You might believe there would be a better use for your RAM than preloading a VM image (I sure would), but apparently that file gets enough use.

Superfetch just isnt *smart* enough to know what kind of files need to be cached or not - it just tries it's best to precache those things that are used most often, and can assume a few things from whether or not you're idle. There is also a very large part of the cache reserved for recent I/O (about 50/50 as far as I can tell), just like the old XP cache.

Still, if you have *constant* disk access, its *not SF*, its something else (you figure that one out). SF loads everything up on start, and then every now and then, it'll switch out to a different set of pages. For example: I got home early one day...so I'm bumming around on my PC. At about the time of day, 7pm, I'd usually fire up BF2142, I noticed that half the cache was suddenly dropped, and it started loading something new. I fired up the monitor, and saw that it was indeed preloading a huge chunk (700mb or so) of BF2142. I've only actually witnessed it changing sets once, so I'm guessing it tries to do it while idle.

The point is...I'm not sure what *pain* is involved. How is that disk access hurting you? It might not be helping you at the moment, but it is based on YOUR usage. Considering that nothing resident is flushed to fill the cache, and the disk access is low priority, what exactly are you losing?

I'd *LOVE* to be able to just mark the folder of a game I'm playing, and tell SF to cache 1gb of it or so. That would be sweet. Maybe in the future, but for what it is right now, its certainly a lot more good than bad.

And on another note, there is absolutely ZERO reason to index WoW, and it certainly isnt doing that by default.

 

[flexy]

great article!!

Btw. i recommend the Patriot Xporter USB stick !

Stick always needs to be same size as RAM, so 4GB of mem means a 4GB stick.

 

[bsobel]

Stick always needs to be same size as RAM, so 4GB of mem means a 4GB stick.

Not sure where you heard that, but this is false.

Bill

 

[Rubycon]

Programs definitely open faster on Vista after a while. Photoshop CS3 opens like MS Word. :Q

On this system I have 8GB RAM and a SAS RAID controller with 2GB cache. Not exactly in need of using ready boost however I have a Sandisk Cruzer 4GB sitting in a drawer. I plugged it in and set Windows to use all 4GB (after removing the useless U3 SW and formatting NTFS) for ready boost. The orange light is flickering a bit so I guess it's working. Every little bit counts.

We have a few Vostro 200's running Vista Enterprise with (gulp!) 1GB. I'm going to try putting a USB drive on one of those too. They don't run bad but they're only used for light stuff too.

 

[JustaGeek]

I haven't read all the posts in this thread, so perhaps my question has been answered before, but...

Is Superfetch adjustable...?

If I add more RAM to the system (I have just replaced my 2x512MB with 2x2GB, for a total of 6GB of RAM), will the superfetch automatically adjust to more physical memory, or will it have to be "reset" (turned off/on...?)

Someone in the Xtreme Systems forums said that if you install Vista with 2GB of RAM, and then add more memory, the Superfetch will still "think" that you have only 2GB of RAM, since that's the amount of memory during installation.

http://www.xtremesystems.org/f...howthread.php?t=182087
"zanzabar: it depends on your motherboard and instal disk version i didnt have a problem with 4GB, if u start with less memory and add more in then it wont use the full 35-50%for supper fetch so i would make sure that u had it all installed"

Is this false?


I see that my computer now uses about 25% of RAM on idle. Should it use more...?

Could it use more...? (to support the philosophy that empty RAM is "wasted RAM".)

Is it normal that my 6GB machine uses "only" 1.5GB on idle...?

TIA

 

[Rubycon]

Well it appears that Server 2008 is Vista without superfetch.

<- posting from Windows 2008 Datacenter server. :Q

 

[SynthDude2001]

Originally posted by: Rubycon
Well it appears that Server 2008 is Vista without superfetch.

<- posting from Windows 2008 Datacenter server. :Q

Superfetch is there (on Server 2008, that is), the service is just disabled by default.

 

[Rubycon]

Originally posted by: SynthDude2001

Superfetch is there (on Server 2008, that is), the service is just disabled by default.

Enabled it but when started I get this.

Could this service really benefit a server?