Hard disk content indexing

edited July 2021 in Chit chat
I just de-indexed content on my Hard-disk mainly as an experiment - but I am still unsure what indexing the content does - I know it speeds it up - does it use a lot of extra space on the HD?
Post edited by dmsmith on

Comments

  • Pretty sure it's used to speed up searches so if you do that a lot it helps.

    No idea how much space it takes up (probably a big hash table).

    It's the sort of thing google does with their web crawlers so their searches are fast.
    Thanked by 1dmsmith
  • Ok thats helpful - thanks. I use a laptop with a CORE i3 and a 128GB SSD, so I prolly would be better with content indexing turned off.
  • It uses up a small amount of disk space. But everything in Windows pretty much relies on the indexing service these days and so turning it off will make everything slower while your hard disk chugs every time you do something like open the start menu.
  • AndyC wrote: »
    It uses up a small amount of disk space. But everything in Windows pretty much relies on the indexing service these days and so turning it off will make everything slower while your hard disk chugs every time you do something like open the start menu.

    The article Timmy provided suggests SSDs are fast enough to have it turned off. I'll just have to try it out and see how my system performs with it on and with it off.
  • You don't really save anything though. The indexing service runs with the lowest CPU priority and the lowest IO priority. If you turn it off, all you are doing is moving the work of refreshing the index to the point it is being queried instead. So turning it off can only ever slow down things you intend to do.
  • AndyC wrote: »
    You don't really save anything though. The indexing service runs with the lowest CPU priority and the lowest IO priority. If you turn it off, all you are doing is moving the work of refreshing the index to the point it is being queried instead. So turning it off can only ever slow down things you intend to do.

    You are absolutely right, it won't save much for the things you said, absolutely. It will save on other things though, fortunately.

    Also, my computer runs a lot faster now, but that's just anecdotal since i also just recently got an SSD.
  • edited July 2021
    I think its because I am not understanding were the benefits come in with having it on. Is it when I am using the search bar to find individual files on my HD (which I rarely do). Or does having content indexing on generally improve performance of the HD when Apps are starting up or accessing files that they need to run?
    Post edited by dmsmith on
  • What I've read on some articles on the internet(!) when I got my SSD, was that turning the indexing off will increase the life of your SSD, especially if you don't search much.

    If you have an HDD (not SSD) the improvement is a bit less.
  • "Turn off indexing" is snake oil. It's the same kind of garbage advice peddled by people who used to advocate registry cleaners, ram doublers and all manner of convoluted "performance" tools that either didn't do anything or actually made matters worse.

    Even if you don't search much, your PC does. When you open the start menu, when you look at control panel and all manner of other places that you might not immediately imagine are searches of your local disks. Turning it off does you no favours at all. It doesn't save memory. It doesn't save CPU. It doesn't save IO. It does introduce random slowdowns occasionally and weird glitches you'll probably never pin down. The best performance advice for Windows you will ever get is to leave it alone instead of following random bad advice on the internet.

    The idea that it will damage SSDs is also nonsense. It's at least vaguely based on facts about life usage of very early SSDs, but the reality is it won't have any effect at all on the lifespan of a modern drive.
  • edited July 2021
    AndyC wrote: »
    "Turn off indexing" is snake oil. It's the same kind of garbage advice peddled by people who used to advocate registry cleaners, ram doublers and all manner of convoluted "performance" tools that either didn't do anything or actually made matters worse.

    Yes, "Turn off indexing" is snake oil.

    Again, this is not about turing off indexing. Indexing happens every time you search and can't be turned off anyway.

    The link I provided earlier doesn't do "turn off indexing".

    What you're talking about is not under discussion. But you're welcome to discuss it further. :)
    Post edited by Timmy on
  • What does indexing do?
    Well, it complies an indexed linked list of files and folders on your mass storage devices, so when anything needs to find a file or folder, rather than having to search each directory/folder from the root directory down until it finds what it is searching for, instead it looks it up in the index.

    This massively cuts down the time it takes to search for a file or folder. It also reduces the amount of wear on a mechanical hard drive. And frees up CPU time during file operations.

    The downside is obviously this index has to be stored somewhere. So it takes up a bit of space. And the OS has to build and maintain this index. But it’s a low priority background task. Hence it’s one of a number of things the computer will do while you are not doing anything very taxing with it.
    Sinclair FAQ Wiki
    Repair Guides. Spanish Hardware site.
    WoS - can't download? Info here...
    former Meulie Spectrum Archive but no longer available :-(
    Spectranet: the TNFS directory thread

    ! Standby alert !
    “There are four lights!”
    Step up to red alert. Sir, are you absolutely sure? It does mean changing the bulb!
    Looking forward to summer in Somerset later in the year :)
  • Timmy wrote: »
    AndyC wrote: »
    "Turn off indexing" is snake oil. It's the same kind of garbage advice peddled by people who used to advocate registry cleaners, ram doublers and all manner of convoluted "performance" tools that either didn't do anything or actually made matters worse.

    Yes, "Turn off indexing" is snake oil.

    Again, this is not about turing off indexing. Indexing happens every time you search and can't be turned off anyway.

    The link I provided earlier doesn't do "turn off indexing".

    What you're talking about is not under discussion. But you're welcome to discuss it further. :)

    The article is literally titled "Turn Off Indexing in Windows for Better Performance'. And yes, literally everything in that article is the same snake oil nonsense that absolutely does not improve the performance of your PC.
  • I have been turning off indexing in Windows and regularly cleaned my registry and junk files for decades, and never witnessed any performance problem whatsoever.

    I cannot say the same for workplace/friends/etc. PCs where the same did not happen ;)

    Snake oil? Yes, thank you :)
  • edited July 2021
    I have an older Asus ROG laptop, HDD not SSD. It was running Win10 somewhat fine up until mid 2018. It became more sluggish with each "critical" update, and lots of little annoyances here and there, and mid 2018 I decided that enough was enough. I barred the OS from fetching any further updates, so on that laptop I'm "stuck" at fall 2018.

    After a few tweaks here and there, the computer runs as nice as it can given that it's Win10.

    And I have to agree with Alessandro ... snakeoil? Please, and thank you!

    I also turned off indexing and that laptop is as spry as it was a decade ago running Win7, albeit it's stunted at fall update '18.

    So, I believe that turning off indexing has its merits but it completely depends on the age of the computer and what flavor of Win10 you're running.
    Post edited by Bloppinius_Blarp on
    [┴ɥᴉs sᴉƃuɐʇnɹǝ ᴉuʇǝuʇᴉouɐllʎ lǝɟʇ qlɐuʞ]
  • I'm a bit behind the times, so I am genuinely interested in hearing what the difference is between "indexing" and a program just looking in the registry to see where its files and folders reside?

    Unless you yourself are looking for a file/folder, why does the OS have to look outside the registry for files and folders?

    Please, I'm honestly interested in understanding the difference.
    1024MAK wrote: »
    What does indexing do?
    Well, it complies an indexed linked list of files and folders on your mass storage devices, so when anything needs to find a file or folder, rather than having to search each directory/folder from the root directory down until it finds what it is searching for, instead it looks it up in the index.[...]

    [┴ɥᴉs sᴉƃuɐʇnɹǝ ᴉuʇǝuʇᴉouɐllʎ lǝɟʇ qlɐuʞ]
  • edited July 2021
    Ok, sorry for all these questions - I had just started exploring my system a bit and came across someone refering to indexing in another discussion. But specifically what i am asking about is not indexing generally, but the check box in properties for my C drive (here an SSD drive) which says allow files on this drive to have contents indexed as well as file properties.

    I mainly wanted to understand what it means by content indexing in addition to file properties?

    Please do what you deem best for your own set up and needs.

    Post edited by dmsmith on
  • Exactly what it sounds like. It scans the contents of files and includes those words in its index, so if you search for a Word document and it turns out you didn't name it well you can still find it based on what was contained in it. Precisely how that works is file type dependent, because file types can install filters for searching within them (so a .sna file could populate the index with a bunch of zxdb metadata etc).

    The overhead of doing this is fairly minimal and so the only real reason to turn it off is if you have an entire drive full of data you're never going to look at or in so esoteric a format that a suitable that the indexer can't look inside them.
  • Ok thanks. I understand it a little better now.. I can see how it would be beneficial in some scenarios to have it on.

    However I just want to have my system arranged so that performance is maximised as I have a slowish i3 mobile processor and I don't do content searches for documents except very rarely.
  • Yes. The thing is, turning it off makes zero difference in performance. All it will ever accomplish is those rare occasions being slower and more annoying.
  • edited July 2021
    But I don't understand how it indexes content - what happens when it gets files that are not text documents? When I de-indexed it spent ages de-indexing LCU folders - I was wondering if it had got stuck in a loop. Its fair enough for developers and people doing a lot of searching.
    Post edited by dmsmith on
  • edited July 2021
    Can you (and/or others) explain my and countless other peoples' experience, please?
    Why do we see a dramatic, positive difference in performance on older machines, turning indexing off?
    AndyC wrote: »
    Yes. The thing is, turning it off makes zero difference in performance.

    Was there a change in how indexing works somewhere along the way? And I'm talking about my older rig stuck at the fall '18 OS update.
    I mean, I rarely use the search function (like once or twice a year, seriously). If I don't use it, then I have no need for it. So, turning it off on my older rig does change the behaviour of the machine. No longer is it indexing here and there dragging the system down.

    But if the OS has (for whatever reason) now been more linked with the indexing/search feature, why was that done? Is MS trying to steer away from using the registry? (see my previous thought in a comment above).

    EDIT: I don't mean to hijack dmsmith's thread but it's definitely related and both he and I are learning from your replies. Thank you! :-)
    Post edited by Bloppinius_Blarp on
    [┴ɥᴉs sᴉƃuɐʇnɹǝ ᴉuʇǝuʇᴉouɐllʎ lǝɟʇ qlɐuʞ]
  • People tend to point the finger at the indexing service because when they install there machine, they see it "doing something" instead of sitting idle and look in the Task Manager and see the indexing service. So that must be slowing everything down, right?

    Except there are a few things that get overlooked. The machine is recently installed, so the index isn't yet built up and the machine is otherwise idle. And it is precisely because the machine is idle that the indexer is kicking in and trying to get work done. If you ignore it and start doing something it immediately backs off and gets out of the way (until the system goes idle again and it steps in to fill the void). It takes a while to fully build the index (because it's doing everything at low priority) and so can give the impression that it is somehow constantly chugging away and always hogging the machine.

    This leads to a slew of articles suggesting you turn things off to speed your computer up. Which in turn feeds the echo chamber of people searching for "why is my Windows install using the disk when I'm not doing anything", finding said articles and repeating the advice verbatim.

    As to why, it's because it's almost always quicker. Indexing data structures are a very well understood and optimised thing at this point (the one in Windows is based on learnings from the SQL Server full text indexing as well as the Bing search engine). They're literally designed around being optimal solutions to this kind of problem unlike on-disk structures (or The Registry) which are more geared around optimising a step-at-a-time traversal of a hierarchy.
    Thanked by 1Bloppinius_Blarp
  • But wait a minute! I had already had that computer setup with Win10 for over two years when I decided to turn off indexing, and it seemingly* got way faster/better. It was no longer chugging along slovenly, and the very distinct, tooooo long of a pause between the computer registering that I was back to the keyboard/mouse and it deciding then to stop indexing is now no longer an issue.

    It felt to me like it was re-doing the index daily, and it was very disrupting for those two years.

    *I put "seemingly" in there because your overall explanation makes sense, new or old computer.
    AndyC wrote: »
    People tend to point the finger at the indexing service because when they install there machine, they see it "doing something" instead of sitting idle and look in the Task Manager and see the indexing service. So that must be slowing everything down, right?

    Except there are a few things that get overlooked. The machine is recently installed, so the index isn't yet built up and the machine is otherwise idle. And it is precisely because the machine is idle that the indexer is kicking in and trying to get work done. If you ignore it and start doing something it immediately backs off and gets out of the way (until the system goes idle again and it steps in to fill the void). It takes a while to fully build the index (because it's doing everything at low priority) and so can give the impression that it is somehow constantly chugging away and always hogging the machine.

    This leads to a slew of articles suggesting you turn things off to speed your computer up. Which in turn feeds the echo chamber of people searching for "why is my Windows install using the disk when I'm not doing anything", finding said articles and repeating the advice verbatim.

    As to why, it's because it's almost always quicker. Indexing data structures are a very well understood and optimised thing at this point (the one in Windows is based on learnings from the SQL Server full text indexing as well as the Bing search engine). They're literally designed around being optimal solutions to this kind of problem unlike on-disk structures (or The Registry) which are more geared around optimising a step-at-a-time traversal of a hierarchy.

    [┴ɥᴉs sᴉƃuɐʇnɹǝ ᴉuʇǝuʇᴉouɐllʎ lǝɟʇ qlɐuʞ]
  • edited July 2021
    BTW on my newer/other PC I have done nothing to change the indexing service. That computer is splendid in this regard, which tells me that newer/more-up-to-date hardware does definitely help. On the other hand, the Asus ROG laptop is from 2008, or thereabouts.

    EDIT: Apologies if I come off as being argumentative. That's not what I mean to do.
    Post edited by Bloppinius_Blarp on
    [┴ɥᴉs sᴉƃuɐʇnɹǝ ᴉuʇǝuʇᴉouɐllʎ lǝɟʇ qlɐuʞ]
  • I don't know how windows does it but what google webcrawlers do is build a map of pairs of words to websites that include them so as to drastically reduce the amount of sites it has to lookup in the first pass.

    So this site will be in a list somewhere on google servers when you search for a phrase with "spectrum world" in it.
  • It usually comes down to something else causing the underlying issue and the indexing service taking the blame (again precisely because it kicks in when you aren't doing anything and so often surfaces in task manager when a user stops working to troubleshoot). There were some notable situations where popular "security" software would mess with NTFS metadata every time they scanned a file, thus helpfully making the file look like it had changed, triggering re-indexing, which then triggered a rescan of the file etc...

    Part of the problem is that people's mental model of how things must work is so far removed from how they actually work in modern PCs that the disconnect gets in the way. We mere humans tend to apply our diagnosis of a problem to our mental understanding of how something must work and when it doesn't work like that at all it's not surprising we come to the wrong conclusions.

    And modern PC diagnostics don't help at all, especially as many of them were designed for a different time. What does 100% CPU usage actually mean in an era when CPU clock speeds change in response to power and heat levels, for example? Is it a proportion of some theoretical maximum? A percentage of the amount of usage compared to idle time? A meaningless number that exists only because we used to show it?
  • Turn off indexing, use Everything, it is literally the first app I install.
    So far, so meh :)
  • edited July 2021
    BTW on my newer/other PC I have done nothing to change the indexing service. That computer is splendid in this regard, which tells me that newer/more-up-to-date hardware does definitely help. On the other hand, the Asus ROG laptop is from 2008, or thereabouts.

    EDIT: Apologies if I come off as being argumentative. That's not what I mean to do.

    You don't come off as being argumentative to me. It is the purpose of the forums to ask questions and debate or offer differing opinions. :-)
    Post edited by dmsmith on
  • CPU usage has always been a bit of a nonsense figure.

    Which core, which pipeline, and which cache?

    In comparison, a Z80A in a ZX Spectrum ALWAYS runs at 100% capacity (at its supplied clock speed). As does the 6502…

    Mark
    Sinclair FAQ Wiki
    Repair Guides. Spanish Hardware site.
    WoS - can't download? Info here...
    former Meulie Spectrum Archive but no longer available :-(
    Spectranet: the TNFS directory thread

    ! Standby alert !
    “There are four lights!”
    Step up to red alert. Sir, are you absolutely sure? It does mean changing the bulb!
    Looking forward to summer in Somerset later in the year :)
Sign In or Register to comment.