Integrating with Everything to add folder sizes to Explorer

[m417z]

Hi, I thought you might find this small project of mine interesting around here.

The native file explorer doesn't show folder sizes, so I decided to see what it takes to add this ability. At first, I created a Windhawk mod which hooks the relevant function and calculates the folder size when it's requested. This approach was obviously slow for folders with a large amount of files (although not as slow as I expected), and several users suggested to do what Everything does, since it's so fast

I didn't feel like implementing a filesystem indexer, but I figured that I can use the Everything SDK, which I happily discovered to exist.

The result:



Check out the mod page for installation and for the source code: Better file sizes in Explorer details.

One thing that could still be improved is that currently, each folder size query triggers an IPC call, which in turn sends the WM_COPYDATA message and waits for a reply. It's not as slow as actually calculating the folder size, but it's slightly noticeable, especially when sorting by size. I'm not sure what can be an alternative which is faster but is still as convenient. It's also not a standard use case - given the Explorer source code, I'd just query all the necessary sizes in one go - so perhaps it's not worth optimizing for. Just sharing in case you have any ideas I didn't think of.

[void]

Thank you for creating Better file sizes in Explorer details.

I am currently working on the IPC for Everything 1.5.
I will add a simple API call to get folder sizes:
EVERYTHING3_USERAPI EVERYTHING3_ULONGLONG EVERYTHING3_API Everything3_GetFolderSizeW(LPCWSTR lpFileName);

Everything 1.5 IPC will use pipes instead of WM_COPYDATA which is much faster (~5x times faster to send all results)
There shouldn't be any delay if you keep the pipe connection alive.

[m417z]

That's great, looking forward to trying the new SDK.

[Kilmatead]

For what it's worth, having recently written a similar plugin for the Xplorer2 file manager, I simplified things slightly by using only the structure definitions from Everything and handling all the IPC myself though a persistent HWND_MESSAGE window but using WaitForSingleObject/SetEvent instead of bothering with GetMessage and all that (since the callback is automatically triggered upon receipt of the WM_COPYDATA reply anyway).

The point is just to say that I got the average query/response time down to below 3 milliseconds per query - which is not exactly dawdling, if you know what I mean, and is almost imperceivable (even when spamming 5000 query/responses when testing).

I too look forward to the 1.5 SDK (pipes are indeed even better), but I might add a small request (@void):

Since the 1.4 SDK doesn't really report an error if you try to query a folder object which isn't currently [size] indexed (such as a reparse-point) it's almost impossible to determine if the zero-size uint64_t return is because the folder size is actually 0 bytes (empty) or if it's simply not indexed (so I can then derive an alternative method). I know the roundabout method of registering reparse points as volumes in Everything itself, but for users who use hundreds of such links - power-users are a grumpy bunch - this method isn't practical.

So the request is a means of knowing if a specific folder object is itself [size] indexed or not, so there's no question about when 0 really means 0.

Thanks. (And not to hijack the thread.)

Also (@m417z): I couldn't tell by looking through your code, but on top of defining "path:wholefilename:" (I used "folder:wfn:" myself), be sure to define pQuery -> search_flags = EVERYTHING_IPC_MATCHDIACRITICS, so that "Pokémon" is always a distinct folder from "Pokemon". Just a thought, as this is not the default behaviour.

[NotNull]

Better file sizes in Explorer details.

a similar plugin for the Xplorer2 file manager

Nice, nice, very nice!
Your craftmanship has been added to the list of software that has Everything integrated, to make it easier to find for other interested people.

EVERYTHING_IPC_MATCHDIACRITICS

Consider also adding case sensitivity to the search queries.
A bit uncommon, but a single folder can contain FILE.txt, file.txt, FOLDER and folder, thanks to support for Windows Subsystem for Linux...

each folder size query triggers an IPC call

Not a developer myself, so the following might be nonsense ...

Instead of querying each folder ( let's say each folder under C:\Windows ) query the parent to list all childfolders, including size.
Using the command-line tool ES.exe, it can be done in one call (see below). Maybe something similar is available using the SDK ....

Code: Select all

T:\>c:\Tools\es.exe   folder:parent:"c:\windows"   -size
    14,614,989 C:\Windows\appcompat
    10,385,866 C:\Windows\apppatch
             0 C:\Windows\AppReadiness
   409,296,772 C:\Windows\assembly
       785,153 C:\Windows\bcastdvr
       574,523 C:\Windows\BitLockerDiscoveryVolumeContents
    71,239,574 C:\Windows\Boot
       104,800 C:\Windows\Branding
       200,498 C:\Windows\BrowserCore
             0 C:\Windows\CbsTemp
    52,503,568 C:\Windows\Containers
            68 C:\Windows\CSC
    11,501,377 C:\Windows\Cursors
         3,430 C:\Windows\debug
     4,482,956 C:\Windows\diagnostics
[...]

[m417z]

For what it's worth, having recently written a similar plugin for the Xplorer2 file manager

Looks great, well done! The code looks way cleaner than mine, my excuse is that I was optimizing for time to release
What's the license of the code? I'd love to borrow some of it if possible, of course not without mentioning you and linking to your plugin post.

The point is just to say that I got the average query/response time down to below 3 milliseconds per query - which is not exactly dawdling, if you know what I mean, and is almost imperceivable (even when spamming 5000 query/responses when testing).

I just did a quick test with my implementation and I got around 15 milliseconds per query. I'll look into adapting this method, and will post back the numbers with the new method on the same hardware.

Also (@m417z): I couldn't tell by looking through your code, but on top of defining "path:wholefilename:" (I used "folder:wfn:" myself), be sure to define pQuery -> search_flags = EVERYTHING_IPC_MATCHDIACRITICS, so that "Pokémon" is always a distinct folder from "Pokemon". Just a thought, as this is not the default behaviour.

Thanks for the tip, I'll add it.

[Kilmatead]

...case sensitivity... ...thanks to support for Windows Subsystem for Linux...

Thanks for that - it's funny no matter how thoroughly you look at something, there's always some aspect in the shadows ready to bite you right back.

Just for the fun of it (and as a proof-of-concept to convince the x2 dev that I wasn't completely crazy), an early version of the plugin actually utilised Everything's FRN: filter which I reckoned was about as targeted as you could get (who needs filenames anyway?), but the drawbacks of overhead-processing and NTFS-only scuppered that as a viable approach. It was a fun idea though - and it worked - but I always imagined with anthropomorphic glee Everything receiving the query and doing a double-take before complying.

(I'm easily amused that way.)

[m417z]

Consider also adding case sensitivity to the search queries.
A bit uncommon, but a single folder can contain FILE.txt, file.txt, FOLDER and folder, thanks to support for Windows Subsystem for Linux...

Good idea, I'll add that too, thanks.

Not a developer myself, so the following might be nonsense ...

Instead of querying each folder ( let's say each folder under C:\Windows ) query the parent to list all childfolders, including size.

That should be possible, but it complicates the implementation, and I decided to keep it simple unless performance becomes a problem and no simpler solution emerges. Querying all parent items also has a downside - when you open a folder in Explorer, and unless it's sorted by size, only the top visible items are queried. So if you e.g. open C:\Windows\WinSxS which has around 20k folders, only around 20 are initially visible. Which one of these is faster - querying 20k items at once or querying 20 items one by one - is a good question, but surely neither is optimal

That's what I meant by "given the Explorer source code, I'd just query all the necessary sizes in one go" - if it was an open source project and not a black box that needs to be reverse engineered, it'd be easier to figure out the exact list of folders that needs to be queried.

[Kilmatead]

What's the license of the code? I'd love to borrow some of it if possible...

No license at all (coding is all about learning, and that's all that matters), so take anything you like, my ego and I don't require any credit. Theoretically once compiled code then comes under the compiling license - or so I'd like to think, thus abjuring me of all responsibility in your potentially evil deeds.

Incidentally, the code was refactored for the recently released c23 revision (-std=c23), so I've been playing around with the GCC 14.2 LLVM 19.1.3 MinGW-w64 toolchain - which includes both clang and gcc compilers, and all the fun that entails. But make no mistake, it's pure C, not C++ (the nullptr keyword might give you a false sense of security). My project is really just a Total Commander plugin crafted for a "third-third-party by way of a fourth".

Just out of interest, what is WindHawk? The site is not awfully clear on that - it seems like a platform that brings shell-extension functionality back into explorer (which Windows so unkindly removed over the years in the name of user "safety"). I'd never heard of it.

and I got around 15 milliseconds per query

To get any kind of reliable times I had to switch to QueryPerformanceCounter, as GetTickCount seems somewhat unreliable under 10ms. I timed from when the query was allocated, filled, and sent to right after the callback return event. I don't have to worry about any display formatting routines, the host takes care of that. (I also went down the whole rabbit-hole of "hey, cool, microseconds! How many of these can I save?" sort of thing. It required rehab and everything, so beware of that demon.)

[Kilmatead]

...and unless it's sorted by size, only the top visible items are queried. So... only around 20 are initially visible.

That's what I meant by "given the Explorer source code, I'd just query all the necessary sizes in one go" - if it was an open source project and not a black box that needs to be reverse engineered, it'd be easier to figure out the exact list of folders that needs to be queried.

You mean the shell does that too? Or is that a WindHawk limitation? I discovered the same (frustrating) thing, but I assumed it was a limitation of the plugin architecture. In my case, the file manager has been extended to specifically call my plugin even for the "unseen" objects which makes my job easier, but only for this project. My other plugins suffer the "Can't see it? Can't process it!" dictum.

[m417z]

No license at all (coding is all about learning, and that's all that matters), so take anything you like, my ego and I don't require any credit. Theoretically once compiled code then comes under the compiling license - or so I'd like to think, thus abjuring me of all responsibility in your potentially evil deeds.

Great, borrowed, the new version is much better in all aspects - code size, stability and performance. Some extra thought was required for making it work in a multi-threaded environment. I hoped to have something elegant and lock-free, but ended up having 2 not-so-elegant mutexes. Oh well.

Regarding performance, quick and rough numbers are around 35 ms per query for the previous version and 15 ms for the new one (I used timeGetTime this time), but there's a large variance. A more end-to-end test is sorting a large folder, which in this test results in an impressive x6 speedup.

BTW one common feedback I received when releasing the first version was: what about Everything 1.5a? I saw that you added instructions in your post to disable the alpha_instance option. I ended up looking for the 1.5a window as a fallback, which is easier for users.

Just out of interest, what is WindHawk? The site is not awfully clear on that - it seems like a platform that brings shell-extension functionality back into explorer (which Windows so unkindly removed over the years in the name of user "safety"). I'd never heard of it.

Windhawk is a platform for creating, sharing and installing mods for Windows. A mod is a snippet of C++ code with special metadata which specifies where the snippet (which is compiled into a dll) is injected. Some APIs are available, such as for function hooking and debug symbol loading. You can think of it as plugins for any program on Windows

A simple example: when you rename a file in Windows and change its extension, Windows thoughtfully asks you for confirmation. There's no option to turn this confirmation off, but with Windhawk, you can just create a mod which injects into explorer.exe, hooks the ShellMessageBoxW function, and returns "yes" without showing the confirmation box. It's around 30 lines of code.

A longer introduction can be found here:
Windhawk, the customization marketplace for Windows programs

The documentation for getting started can be found here.

BTW Windhawk uses the same LLVM MinGW toolchain

...and unless it's sorted by size, only the top visible items are queried. So... only around 20 are initially visible.
...

You mean the shell does that too? Or is that a WindHawk limitation?

The GetSize function is called lazily once items are brought into view, and in my case it's a good thing, as the IPC is only fired for visible items. Sorting is more wild, GetSize is called multiple times per file as part of the sorting. I had to add caching for that, otherwise sorting becomes really slow.

[Kilmatead]

Which one of these is faster - querying 20k items at once or querying 20 items one by one - is a good question, but surely neither is optimal

This question kind of struck an amusing chord in my mind, so I did a simple test:

First, as a vague benchmark (it's awkward to create a separate timer thread just for this), I just roughly stopwatched a browsing into C:\Windows\WinSxS which on my system contains 16,400 folders, and letting my plugin query them consecutively took around 41 seconds, which essentially amounts to 41,000 milliseconds / 16,400 == 2.5ms, which is about what I expected given the margin for error.

It did, however, feel like clubbing a baby seal over the head; I guess the plugin passes the robust-ness test.

So setting up a specialised query of parent:"C:\Windows\WinSxS" (collecting not just size but also path/name information since you'd have to sort it all out yourself later), Everything took (on my system) 10ms to return a 4MB chunk of WM_COPYDATA, with the relevant goodies inside. It would take a hilarious amount of void's pointer-math leapfrogging to sort this into bitesize pieces for you to munch on, but you get the idea.

The answer to your somewhat rhetorical question is 41,000 milliseconds vs. 10 milliseconds (plus a lot of sorting, which probably would take more time to think about than to actually execute).

Not sure if it's worth the trouble or not, but there you go.

Great, borrowed, the new version is much better in all aspects - code size, stability and performance.

Hah, I've been chop-sueyed, spliced, daggered, and immortalised in code all at the same time - that was actually fun to read! Thanks for that. Glad it helped.

Incidentally, did you know that an expression of beauty like:

Code: Select all

g_gsReply.liSize = *(int64_t *)(((BYTE *)(list + 1)) + sizeof(EVERYTHING_IPC_ITEM2));

...really just ends up as a call to memcpy in the final ASM? Obviously you know that, of course, I'm just ranting - I found it rather depressing, but predictable. I like my way better. (Though I'm miffed you prefer BYTE to the more elegant byte_t notation - to each their own!)

And thanks for the info on WindHawk - worth a look, for sure. Odd that I never heard of it before, but first time for everything.

Incidentally I've been following that mstorsjo / llvm-mingw project for awhile now, and it gets updates every couple of weeks (if you're into bleeding-edge LLVM stuff). Most people don't update their compilers that often, but it's nice the world keeps churning out toys for us muggles to mess around with.

[m417z]

The answer to your somewhat rhetorical question is 41,000 milliseconds vs. 10 milliseconds (plus a lot of sorting, which probably would take more time to think about than to actually execute).

Not sure if it's worth the trouble or not, but there you go.

Great analysis, Everything is truly impressive! Now it's more tempting to do it, but due to the "void's pointer-math leapfrogging" and the fact that it's not a perceptible problem in my case unless sorting is used, I'll leave it for another day.

A problem that I'm more eager to solve is that when sorting by size, Explorer keeps files and folders separated. It does that not only for size, but for other fields such as modified date too. For size, it'd be much better to not have this separation.

Incidentally, did you know that an expression of beauty like:

Code: Select all

g_gsReply.liSize = *(int64_t *)(((BYTE *)(list + 1)) + sizeof(EVERYTHING_IPC_ITEM2));

...really just ends up as a call to memcpy in the final ASM? Obviously you know that, of course, I'm just ranting - I found it rather depressing, but predictable. I like my way better.

Here's what I see:

Code: Select all

    ; if (list->numitems) {
    ;     g_gsReply.liSize =
    ;         *(int64_t*)(((BYTE*)(list + 1)) +
    ;                     sizeof(EVERYTHING_IPC_ITEM2));
    ;     g_gsReply.bResult = true;
    ; } else {
    ;     g_gsReply.liSize = 0;
    ;     g_gsReply.bResult = false;
    ; }

    cmp dword ptr ds:[rax+0x04], 0x00
    jz short @L00000002
    mov rax, qword ptr ds:[rax+0x1C]
    mov cl, 0x01
    jmp short @L00000003
    ; ...

@L00000002:
    xor eax, eax
    xor ecx, ecx

@L00000003:
    mov qword ptr ds:[$$0F0D8], rax
    mov byte ptr ds:[$$0F0D4], cl

Also, what's "my way"? Isn't it the same? Perhaps I'm missing something.

I like my way better. (Though I'm miffed you prefer BYTE to the more elegant byte_t notation - to each their own!)

Well, it saves me a typedef. And it's more consistent with the rest of the code that's already saturated with WinAPI types.

If anything, since I'm already a sinner who uses C++, I'd go with std::byte for a more confirming code. And in C, I'd probably go with the standard uint8_t.

Edit: hmm, turns out that std::byte and unsigned char are special, while std::uint8_t isn't. TIL.

If we're into style review, I'd drop the L in LPSTR, and add a C where it's const. const is nice, and it's in C way before nullptr, and now in C23 they even added type-generic string functions (e.g. strstr, const in -> const out).

And thanks for the info on WindHawk - worth a look, for sure. Odd that I never heard of it before, but first time for everything.

Yeah, well, Windhawk isn't as popular as Everything. You're welcome! You have the vibes of somebody who's going to enjoy it

[Kilmatead]

Also, what's "my way"? Isn't it the same? Perhaps I'm missing something.

No, you're not missing anything, like I said, I'm just raving (and possibly not looking as pedantically as I should at what I write) - bloody optimising compilers take all the fun out of everything. Wait until someone shoves the contemporaneous pseudo-AI craze into it - then we're really doomed. I first learned C about 45 years ago when I was 12, and I've never left it, but sometimes it feels like it's trying to leave me. (I was a philosophy major in University, so these things hang heavily over my head, for no good reason.)

If we're into style review, I'd drop the L in LPSTR, and add a C where it's const. const is nice, and it's in C way before nullptr, and now in C23 they even added type-generic string functions (e.g. strstr, const in -> const out).

And it's more consistent with the rest of the code that's already saturated with WinAPI types.

Yeah, but you left the int64_t instead of going all LONGLONG on us. All this fuss over 8 little bytes.

I just can't let it rest: Some things are worth the price of a typedef! But yeah, if I was a more considerate coder, I would be far more diligent about using const where appropriate, but... they even "added" peremptory support for throwing exceptions into c... whatever happened to flying by the seat of your pants and living dangerously? I feel like yelling at the kids to get off my lawn.

Anyway, this was all a success... now we get to look forward to some rewrites whenever the 1.5 SDK drops...

[NotNull]

A problem that I'm more eager to solve is that when sorting by size, Explorer keeps files and folders separated. It does that not only for size, but for other fields such as modified date too. For size, it'd be much better to not have this separation.

Everything 1.5 has the possibility to mix files and folders when sorted by one of the indexed properties, like Name, Size, Path.
(see Menu => Tools => Options => Indexes to find out what is currently indexed).
Mixing files and folders is not available in version 1.4 due to different database structure.

The current SDK is based on Everything 1.4 and will be updated soon to support 1.5 and likely support mixed sorting too.
As a workaround for now, add mix-sort: to the search query. Example:
(note the postion of hiberfil.sys and pagefile.sys amidst the folders)

Code: Select all

T:\>c:\tools\es.exe   parent:c:\   mixsort:   sort:size  -size
27,679,114,967 C:\Windows
13,210,692,522 C:\Media
 9,386,144,284 C:\_DL
 7,038,909,110 C:\$Recycle.Bin
 6,745,325,568 C:\hiberfil.sys
 5,353,883,321 C:\Program Files
 4,510,073,744 C:\Program Files (x86)
 2,550,136,832 C:\pagefile.sys
 1,891,948,209 C:\Users
 1,603,295,091 C:\ProgramData
   691,650,898 C:\Tools
   123,974,165 C:\2install
    45,402,982 C:\Apps
    16,777,216 C:\swapfile.sys
        20,568 C:\System Volume Information
        12,288 C:\DumpStack.log.tmp
         4,660 C:\temp
         1,109 C:\Recovery
             0 C:\PerfLogs
             0 C:\Intel

it's not a perceptible problem in my case

Indeed; there are not a lot of folders with more than 100 subfolders. And most of those are not very "inhabitable" and will likely be visited sporadically, if ever ...

BTW: Type or paste the following search-query in Everything 1.5 to list the amount of subfolders per folder:

Code: Select all

folder:   add-column:childfoldercount   sort:childfoldercount-descending

[m417z]

Mixing files and folders is not available in version 1.4 due to different database structure.
...
As a workaround for now, add mix-sort: to the search query.

The limitation here is Explorer, not Everything. It receives the sizes and decides how to sort the list.

it's not a perceptible problem in my case

Indeed; there are not a lot of folders with more than 100 subfolders.

What I meant is that if the list isn't sorted by size, only the top visible folders are queried for size, so the speed penalty is limited by the monitor size, not by the amount of folders in the list. At least not until the user begins to scroll, at which point it's not too challenging to keep up with that.

[NotNull]

What I meant ...

Yes, I got that. The comment was about when sorting these folders by size.
The folders that have > 100 childfolders are likely not visited frequently, which mitigates the issue largely.

The limitation here is Explorer, not Everything. It receives the sizes and decides how to sort the list.

OK, missed that ..

[NotNull]

Anyway, this was all a success... now we get to look forward to some rewrites whenever the 1.5 SDK drops...

If you get bored looking forward (doubt that), a wild suggestion:

WizTree (disk space analyzer) and Directory Opus (file manager) both can show relative size / "Percent of Parent". Implementation is slightly different between the two. See screenshot below.

Would be cool if explorer and xplorer2 could do something similar, now they both are able to "speed-read" foldersizes

Percent of Parent.jpg (118.32 KiB) Viewed 1431 times

[Kilmatead]

...only the top visible folders are queried for size, so the speed penalty is limited by the monitor size, not by the amount of folders in the list. At least not until the user begins to scroll, at which point it's not too challenging to keep up with that.

I said this before, but I really am amazed that this seems to be the way it works - I thought I was the only one fighting it. I find it hard to believe MS somehow imagined Explorer as the original JIT paradigm...

there are not a lot of folders with more than 100 subfolders. And most of those are not very "inhabitable" and will likely be visited sporadically, if ever ...

I imagine there's more than a few people with oversized (and over-organised) music and photo collections which would beg to differ. (Just joking...)

[Kilmatead]

Would be cool if explorer and xplorer2 could do something similar, now they both are able to "speed-read" foldersizes

Doing that numerically is easy enough with yet another plugin, but as seen above, the whole "not processed until scrolled-through" creates a problem for sorting (which is what most people, I imagine, would like that kind of forensic information for (x2 has the usual pie-chart stuff for that). Sometimes fighting with the dev's of these programmes to get (what they tend to refer to as "fetishistic interests") fixed, is a losing proposition, God knows.

Not all developers are as open as void. No pun intended.

Not to mention, because of you I now have to install this linux subsystem thing just to see what kind of extra wrenches it may be tossing into the works that I haven't thought of yet... (joking again) forward-slashes may be another Y2K problem in the making for young and innocent parser algorithms...

[NotNull]

No need to install WSL:

Code: Select all

T:\>md demo

T:\>fsutil file setcasesensitiveinfo t:\demo
Case sensitive attribute on directory t:\demo is enabled.

T:\>cd demo

T:\demo>echo.> file.txt

T:\demo>echo.> FILE.txt

T:\demo>dir
 Volume in drive T is RAMDISK
 Volume Serial Number is C050-3B28

 Directory of T:\demo

09-11-2024  00:10    <DIR>          .
09-11-2024  00:10                 2 FILE.txt
09-11-2024  00:10                 2 file.txt
               2 File(s)              4 bytes
               1 Dir(s)     737.603.584 bytes free

T:\demo>

[Kilmatead]

No need to install WSL...

It's not that I didn't believe you (of course you're right), it's more what sparked my curiosity about other kinds of unintended things that I might find in there that could cause the end of the world. That's what I do, I mess with stuff until it breaks, so I can fix it to mess with other stuff... it's a lifelong occupation.

(Though I am disappointed it doesn't seem to employ forward slashes...?)

[m417z]

I did it!

BTW: Type or paste the following search-query in Everything 1.5 to list the amount of subfolders per folder:

Code: Select all

folder:   add-column:childfoldercount   sort:childfoldercount-descending

I skimmed over it too quickly last time, that's really slick!

[m417z]

@Kilmatead looks like you missed the ChangeWindowMessageFilterEx call that exists in the official SDK. I missed it too until a user reported that elevated programs get a timeout (at least there's a timeout now!). Maybe it makes less sense running x2 elevated, but there might be valid use cases, and all it takes to support it is a single call.

[Kilmatead]

looks like you missed the ChangeWindowMessageFilterEx call that exists in the official SDK.

I hadn't missed it, I was just under the impression that didn't apply to messages below WM_USER (that's not a great authoritative link, but I can't find the one I was going by right now). (0x004A < 0x0400)


ChangeWindowMessageFilterEx

Certain messages whose value is smaller than WM_USER are required to be passed through the filter, regardless of the filter setting. There will be no effect when you attempt to use this function to allow or block such messages.

It's the vague "certain" part of that... not a good thing to do with documentation.

Just in case, I did all sorts of tests against elevated/non-elevated processes and I never received an access failure on full UAC so I didn't worry about it (nor, have I had any user-problems with the plugin). Maybe it has more to do with your actual hooking of the explorer process (which is more invasive)?

However, if it relieves even a possible problem, you're right, a single call is a single call.

What, though, does "elevated programs get a timeout" mean? Never heard of that one. I used SendMessageTimeoutW more in an abundance of caution (against process hangs) over actual causation, but that shouldn't apply to UIPI.

There's also the small matter that MessageFilter might need LoadLibraryW(L"user32.dll") for XP support (which is also in the SDK), but I was hoping that was too legacy... maybe it isn't... except XP doesn't suffer from UIPI. What?

[m417z]

It's the vague "certain" part of that...

Probably a better word would be "selected" messages. Here's some research on the topic.

Just in case, I did all sorts of tests against elevated/non-elevated processes and I never received an access failure on full UAC so I didn't worry about it (nor, have I had any user-problems with the plugin). Maybe it has more to do with your actual hooking of the explorer process (which is more invasive)?

I don't think so. I'd expect that if you'll run Everything as non-elevated, and x2 as elevated, you'll encounter this problem. Does it just work for you?

What, though, does "elevated programs get a timeout" mean? Never heard of that one. I used SendMessageTimeoutW more in an abundance of caution (against process hangs) over actual causation, but that shouldn't apply to UIPI.

SendMessageTimeoutW works, because elevated->non elevated is not a problem, but when Everything tries to reply, that doesn't work. The query then times out in WaitForSingleObject.

There's also the small matter that MessageFilter might need LoadLibraryW(L"user32.dll") for XP support (which is also in the SDK), but I was hoping that was too legacy... maybe it isn't... except XP doesn't suffer from UIPI. What?

IMO it's time to let XP go. Latest MSVC doesn't even have an option to produce executables which run on it. In my case, Windhawk only supports Win7 and up anyway, and it might change to Win10/Win11-only in the not-so-distant future.

[Kilmatead]

Here's some research on the topic.

Ah yes, I remember reading that a few years back... well, most of it anyway. Between marshalling protocols and exactly what the manifest dictates, my eyes started to wander...

I'd expect that if you'll run Everything as non-elevated, and x2 as elevated, you'll encounter this problem. Does it just work for you?

Exactly what I thought was supposed to happen, but I could never get it to fail, so I gave up and decided to pursue it only if people complained (no one did - but possibly no one uses my stuff?). It's a pain to create user accounts and change elevations and restart the machine 4 or 5 times in an afternoon just to test a hypothesis I could never prove or disprove. It's also not impossible that I've so messed up my security setup over the years that I managed to change something permanently which I shouldn't have, so my testing might be meaningless.

I'm also not a "real" developer (in the sense of anyone having a monetary stake in me), so maybe people aren't worried enough to complain directly to me. They do about bugs though... those complaints I hear. And requests. Always requests.

The query then times out in WaitForSingleObject.

Did you encounter this? That's what it's there for, after all... but still, I keep wondering about exactly how much of a DLL's self-created window is actually under the aegis of the process's messaging system - especially for a window which is created explicitly for and dedicated to messaging. I also doubt the danger of errant WM_COPYDATA messages... if a process isn't set up for them, I assume (always a bad idea!) that DefWindowProcW would just delete them and deallocate their resources and be done with it. The WinAPI is a strange place, though, so one never knows.

IMO it's time to let XP go. Latest MSVC doesn't even have an option to produce executables which run on it.

Yeah I use the compiler targeted for MSVCRT simply because x2 still supports those older machines, so it's not quite time to fully embrace the UCRT (if it was, no one would bother creating MSVCRT builds!).

I did it!

I assume the solution is in the latest Windhawk link, so I should read it again? Is this the "scroll to sort" problem?

[m417z]

I'd expect that if you'll run Everything as non-elevated, and x2 as elevated, you'll encounter this problem. Does it just work for you?

...
It's also not impossible that I've so messed up my security setup over the years that I managed to change something permanently which I shouldn't have, so my testing might be meaningless.

Yeah, that can explain it, otherwise I'd expect it to be easily reproducible.

The query then times out in WaitForSingleObject.

Did you encounter this?

Yes, that's what I'm seeing, it ends up failing with ES_QUERY_REPLY_TIMEOUT.

I did it!

I assume the solution is in the latest Windhawk link, so I should read it again? Is this the "scroll to sort" problem?

When sorting by size, files end up in one separate chunk, and folders in another (see screenshot below). That's the default Explorer behavior, which also applies when sorting by other columns. I added an option changes sorting by size to disable this separation.

You can see the changelog on the mod page. Each version also links to the code on GitHub, where you can also see the code history and diffs.

It's no longer a problem related to querying sizes or to Everything, it's just making Explorer behave in a way that makes more sense after it got the sizes.

[Kilmatead]

Yes, that's what I'm seeing, it ends up failing with ES_QUERY_REPLY_TIMEOUT.

Fair enough - I can hardly argue with my own error message. God knows that's enough said on the subject.

When sorting by size, files end up in one separate chunk, and folders in another...

Ok, bear in mind I'm asking this before perusing the code (it's the end of a day of manual labour and I need a shower and sustenance before else), but if I recall correctly, your code receives the objects initially as PIDLs, no? Surely you're not querying non-folders for size via Everything when you can just do it natively at a fraction of the cost.

Everything is absolutely wonderful for folder-sizes, but (assuming you still have that curious 15 millisecond handicap), files would just slow you down. That's how my plugin works - files are never even submitted for consideration.

If, on the other hand, I'm wildly mistaken and am misremembering the code I reviewed the first time, feel free to turn up your expensive collar and ignore this poor pretender's protestations of petition.

[m417z]

When sorting by size, files end up in one separate chunk, and folders in another...

... if I recall correctly, your code receives the objects initially as PIDLs, no? Surely you're not querying non-folders for size via Everything when you can just do it natively at a fraction of the cost.

Essentially, I call the original GetSize function, and proceed with querying Everything only in case of an empty result. In practice, in most (all?) cases it means that files are handled by Explorer, and folders are handled by the mod. If at some point in the future Windows learns to query folder sizes, it should bypass Everything altogether.

[m417z]

A user reached out to me with an interesting problem. Can you spot the root cause?
https://streamable.com/m631xz

From a quick analysis, here's what's going on:

• Everything opens the folder by using SHOpenFolderAndSelectItems
• Explorer opens the folder and lists the contained files and folders
• For each folder, the mod kicks in and queries Everything for the folder size
• Meanwhile, Everything is blocked in SHOpenFolderAndSelectItems, which for some reason seems to return only after the list is fully loaded
• The mod times out, resulting in a long delay and a lack of folder sizes
• Afterwards, refresh works fine as Everything is no longer blocked

On Everything's side, a workaround can be to either run explorer.exe explicitly or to run SHOpenFolderAndSelectItems in another thread. Neither is pretty, but also not horrible. An upside is that regardless of the mod, opening a folder won't block Everything, but I don't think anyone ever complained about that.

On the mod side, I can do the same for Everything.exe by hooking SHOpenFolderAndSelectItems which is the short-term solution I have in mind right now. The mod injects into all processes, including Everything, anyway.

Any better ideas?

BTW interestingly, the problem always happens with the old ribbon UI of Explorer (what's on the video), but only rarely happens with the new UI (on the other screenshots in this thread). Perhaps they changed some of the code to become more asynchronous.

[Kilmatead]

The mod times out, resulting in a long delay and a lack of folder sizes

What part of the mod times out - outgoing, or incoming? Or this this related to your mutexes? This aspect of hooking is your bailiwick, not mine - unless void can chime in more knowledgeably. I can reproduce it though, ribbon or no ribbon, so something's odd.

On a (probably) completely unrelated note, your mod seems to inexplicably fail on the "C:\Windows\assembly" folder - when viewed simply browsing to C:\Windows itself (all other folders surrounding it populate correctly), which is curious because ES doesn't have a problem retrieving its size (855MB for me), if I query it directly (my mod). To be fair, that folder (or at least its contents) are "different", but there's no reason to browse into it to query its size and a refresh does not help, in this case.

It's the only other place I've seen your mod fail to print anything (blank, as opposed to 0 bytes).

[m417z]

The mod times out, resulting in a long delay and a lack of folder sizes

What part of the mod times out - outgoing, or incoming? Or this this related to your mutexes? This aspect of hooking is your bailiwick, not mine - unless void can chime in more knowledgeably. I can reproduce it though, ribbon or no ribbon, so something's odd.

Incoming - ES_QUERY_REPLY_TIMEOUT. SendMessageTimeout succeeds, but WaitForSingleObject times out since WM_COPYDATA doesn't arrive until the whole list is loaded and Everything is no longer blocked.

I created a simple test mod which hooks SHOpenFolderAndSelectItems and runs it in a new thread. That fixes the problem. I'll look into integrating it with the main mod. To try it, open Windhawk, click the bottom right button to create a new mod, replace the code with the test mod code, click "Compile Mod", then "Exit Editing Mode".

On a (probably) completely unrelated note, your mod seems to inexplicably fail on the "C:\Windows\assembly" folder - when viewed simply browsing to C:\Windows itself (all other folders surrounding it populate correctly), which is curious because ES doesn't have a problem retrieving its size (855MB for me), if I query it directly (my mod). To be fair, that folder (or at least its contents) are "different", but there's no reason to browse into it to query its size and a refresh does not help, in this case.

It's the only other place I've seen your mod fail to print anything (blank, as opposed to 0 bytes).

I can't reproduce it, not on my real computer and not in a VM. Can you post debug logs?

• Go to the mod's Advanced tab.
• Set Debug logging to Mod logs.
• Click on Show log output.
• Open C:\Windows, make sure the assembly folder is visible.
• Set Debug logging back to None, grab the logs.

I'd expect a log line saying "Failed to get size: ...".

[Kilmatead]

Not most detailed of logs (?) - there's just a single "Failed: 80004002", but not much else out of place (this is just the relevant fraction):

Code: Select all

11:35:11.603 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [639:CFSFolder__GetSize_Hook]: >
11:35:11.605 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [694:CFSFolder__GetSize_Hook]: Done: 0
11:35:11.605 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.606 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.606 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.607 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.607 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [639:CFSFolder__GetSize_Hook]: >
11:35:11.610 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [665:CFSFolder__GetSize_Hook]: Failed: 80004002
11:35:11.610 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.610 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.610 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.611 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [639:CFSFolder__GetSize_Hook]: >
11:35:11.613 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [694:CFSFolder__GetSize_Hook]: Done: 785153
11:35:11.614 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.614 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.614 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.614 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.615 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [639:CFSFolder__GetSize_Hook]: >
11:35:11.618 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [694:CFSFolder__GetSize_Hook]: Done: 69937630

The "empty" bit survives a machine restart and all, it's consistently failing just in that one folder.

[m417z]

Not most detailed of logs (?) - there's just a single "Failed: 80004002", but not much else out of place (this is just the relevant fraction):
...

Seems like this fails with E_NOINTERFACE:

Code: Select all

winrt::com_ptr<IShellFolder2> childFolder;
hr = shellFolder2->BindToObject(itemidChild, nullptr,
                                IID_PPV_ARGS(childFolder.put()));

No idea why. If you want to investigate it, maybe you can create a regular program which does something similar and see if you get a similar error.

[void]

Since the 1.4 SDK doesn't really report an error if you try to query a folder object which isn't currently [size] indexed (such as a reparse-point) it's almost impossible to determine if the zero-size uint64_t return is because the folder size is actually 0 bytes (empty) or if it's simply not indexed (so I can then derive an alternative method). I know the roundabout method of registering reparse points as volumes in Everything itself, but for users who use hundreds of such links - power-users are a grumpy bunch - this method isn't practical.

Everything calculates the folder size from all the child files and grandchild files in your index.
Everything doesn't follow folder junctions with NTFS parsing.
Everything will report folder junctions with a size of 0.

The SDK will return UINT64_MAX (0xffffffffffffffff) if folder size indexing is disabled.

One idea is to check the folder for the FILE_ATTRIBUTE_REPARSE_POINT (0x00000400) flag and handle it differently.

Also (@m417z): I couldn't tell by looking through your code, but on top of defining "path:wholefilename:" (I used "folder:wfn:" myself), be sure to define pQuery -> search_flags = EVERYTHING_IPC_MATCHDIACRITICS, so that "Pokémon" is always a distinct folder from "Pokemon". Just a thought, as this is not the default behaviour.

Consider also adding case sensitivity to the search queries.

The new Everything3_GetFolderSizeW API will lookup folders by case first.
A case insensitive search is performed if a folder is not found.
Diacritics are always matched.
You could emulate something similar with Everything 1.4.

Meanwhile, Everything is blocked in SHOpenFolderAndSelectItems, which for some reason seems to return only after the list is fully loaded

I have put on my TODO list to move SHOpenFolderAndSelectItems into a thread.
SHOpenFolderAndSelectItems doesn't return until the list is loaded.
Can you load the folder sizes after the list is loaded?

[m417z]

Meanwhile, Everything is blocked in SHOpenFolderAndSelectItems, which for some reason seems to return only after the list is fully loaded

I have put on my TODO list to move SHOpenFolderAndSelectItems into a thread.
SHOpenFolderAndSelectItems doesn't return until the list is loaded.
Can you load the folder sizes after the list is loaded?

Not easily. It's easier for me to just hook SHOpenFolderAndSelectItems and run it in a new thread. I'll make an extra check that I'm running in the thread that owns the Everything main window, so that once Everything starts doing that too, the mod won't spawn yet another thread.