Modification date of a folder is not updated in the index after deletion of a file inside

[aleksusklim]

When a file is deleted (or moved away) from any directory – Everything fails to update in its database the modification timestamp of the parent directory of that file. But the folder timestamp was actually changed, which can be seen from Explorer or by forcing Everything to rebuild its database.

Some cases when the modification date of a directory is updated are listed here:
https://stackoverflow.com/a/46428666

Steps to reproduce

1) Open some non-empty folder on an NTFS disk that is monitored by Everything.
2) Check that folder's modification date and time, both with Explorer and with Everything.
3) Delete any file in that folder (in Recycle Bin or permanently).
4) Notice that Explorer shows updated timestamp for that folder.
5) Try to seek for it in Everything: for example, set Filter to "Folder" and sort by modification date (as we know that this folder ought to be the newest).
6) You will see that folder modification date in Everything index is not updated.
7) Go to Options>Indexes and do "Force Rebuild".
8) Now Everything will properly show this folder as the most recent element (probably except for the Recycler Bin itself) with modification date matching the time of file deletion.

Illustration

I have a folder "B:\Test" with file "file_1.txt" and subdirectory "Folder", that contains "file_2.txt". The local disk is monitored both by Everything 1.4.1.1009 x64 and Everything 1.5.0.1289a x64:


I deleted "file_2.txt" with Explorer to Recycle Bin behind the scene:
Notice that the bin itself is updated to 12:42, but the Folder is still at 12:39.


Then I reverted the deletion by Ctrl+Z / Undo in Explorer:
Now the Folder gets updated to 12:43, which means that an addition of a file is detected correctly. (In the bottom window, the modification date is listed in the one from the last column)


Finally, I tried to rename "file_2.txt" to "file_2_renamed.txt":
Here, the 1.5-alpha version detected the change and shows 12:45, while the 1.4-stable version still shows 12:43.


Background

I wanted to use Everything API (either as SDK or by calling es.exe) to reliably detect the fact that nothing was changed under the particular path. Then in case of a most recent modification date mismatch (on a file or a folder) my program will do a more advanced checks to find out, are there any real changes happened.

But now I faced the fact that a sole deletion of a file gets undetected and passes my test of checking modification timestamps, despite the actual parent timestamp was updated to the time of a child file deletion moment.

Versions of Everything tested

"1.4.1.1009 (x64)" – detects only file additions, but not renames and deletions;
"1.5.0.1289a (x64)" – detects additions and renames, but still not deletions.

Looks like subfolder (not sub-file) deletions and renames are detected correctly by both versions.

[void]

Thanks for the detailed bug report aleksusklim,

It's a synchronization issue.

Everything is notified about the file deletion before the parent folder timestamp is updated.



I'll experiment with using the timestamp from the USN record for the parent date modified instead.
From brief testing it appears to work.

I will get back to you..

[aleksusklim]

Ah, a race condition! Yeah, that happens.
Thank you for your reply.

May I ask you about your current plans and state of adding the Index Journal to API SDK?
I saw "add journal api to SDK." in your TODO list rather long ago, and I wonder, what is its current status?

I wanted to create an intelligent backup solution, that will monitor changes on disk in realtime. Then, for each modified object it will decide what to do: either copy it immediately, or post-pone the copying after some timeout (to let the modifying program finish its work) making intermediate copies if needed.
If the backup destination disk is not connected, my program will store the list of files to backup later, and will do it when the target will be online. With the program like this, a user will need only select some source paths and backup paths to let it handle the constant low-cost (in terms of CPU resources and disk activity) backuping of everything there (maybe with some filters for file types and filesize).

I planned to add even more features later, like storing the deferential copy of each file, to be able to restore any previous state (optionally deleting old states automatically), compressing and deduping.

The main challenge is to detect the modifications on source disk. My first thought was to ask Everything about last modification date of recent files and folders, but that will not give me renames and movings (thus, my program will need to copy all files in a renamed folder, instead of move or symlink their backups).
Then I wanted to parse MFT and USN by myself, but that stuff is too low-level for me, because in fact I will have to reinvent Everything from scratch.

But when I saw your Index Journal feature, I realized that it will give me exactly what I need, so I could focus directly on optimizing the backup process itself, rather that tracking source changes!

And since I saw your plans to add Index Journal access to SDK, I decided to just wait for this feature.
Now I ask you, if you could implement it sooner (but not exactly right now; the next year is fine), then I will actually use it in my future program (which I didn't start to write yet), reporting any bugs I encounter.

[void]

The Index Journal API is still on my TODO list.

For now, consider enabling index journal logging and parsing the logs.

[aleksusklim]

A quick check reveals that Everything is not flushing the journal file to disk immediately.

I made some changes on source disk, I can see them in "Menu > Index > Index Journal …", but file "index-journal-2021-12-15-1.5a.txt" stays 0 bytes.
Only when I exited Everything, that file was actually written.

Is there any other setting that I could set to force it flush sooner?

[void]

Unfortunately, the Index Journal Log is buffered in memory.

The log will use a different filename for each day.
It would only be practical to parse the logs from the previous day.

[void]

Everything 1.5.0.1290a will now update parent folders with the correct timestamp.

I have started working on an Index Journal API.

I have also added a update_parent_date_modified_for_events* ini setting.
(* this should have been update_parent_date_modified_for_all_events)
This might be useful for tracking folders where the child file content is changed.

[aleksusklim]

Looks like it's working! Thank you for the fix.

I checked 1.5.0.1290a with ES-1.1.0.21, and now all folder modifications are detected instantly.

Regarding update_parent_date_modified_for_events: as I understood, it updates parent timestamp only in Everything database, but not on disk. Thus, if the database would be rebuilt, those updated timestamps will be lost.
I decided to ignore this behavior (and always invoke two requests: for files and for folders) to make my queries more reliable.

Now I have a small question about documentation of "dm:"-query syntax in es.exe.
Currently it states:

dm:<date> Search for files and folders with the specified date modified.
...
function:>value Greater than value.
...
Date Syntax:
year
month/year or year/month depending on locale settings
day/month/year, month/day/year or year/month/day depending on locale settings
YYYY[-MM[-DD[Thh[:mm[:ss[.sss]]]]]]
YYYYMM[DD[Thh[mm[ss[.sss]]]]]
...
-date-format <format>
0=auto, 1=ISO-8601, 2=FILETIME, 3=ISO-8601(UTC)

– It never states that "FILETIME" format is acceptable for searches too. For example:

Code: Select all

es.exe -instance 1.5a -sort date-modified-descending /o-d -dm -csv -no-header -path "C:\Test\" -date-format 2

132851487259089697,"C:\Test\Folder_1\File_4.txt"
132850837225247991,"C:\Test\File_3.txt"
132851495760670366,"C:\Test\Folder_1"
132851492644565250,"C:\Test\Folder_2"

Then I can seek by it:

Code: Select all

es.exe -instance 1.5a -s -txt -path "C:\Test\" dm:^>132851492644565250
C:\Test\Folder_1

Here comes another caveat: format accepts either dm:^>X or dm:>"X", but not "dm:>X"
I mean, full quoting will not work:

Code: Select all

es.exe -instance 1.5a -s -txt -path C:\Test\ "dm:>132851492644565250"

es.exe -instance 1.5a -s -txt -path C:\Test\ dm":>132851492644565250"

es.exe -instance 1.5a -s -txt -path C:\Test\ dm:">132851492644565250"
C:\Test\Folder_1

es.exe -instance 1.5a -s -txt -path C:\Test\ dm:">"132851492644565250
C:\Test\Folder_1

Maybe it's worth to mention this behavior somewhere near to

Use ^ to escape \, &, |, >, < and ^.

Or this could be a bug, since I don't understand why I shouldn't put in quotes the whole argument.

[void]

Thank you for your reply aleksusklim,

Regarding update_parent_date_modified_for_events: as I understood, it updates parent timestamp only in Everything database, but not on disk. Thus, if the database would be rebuilt, those updated timestamps will be lost.

update_parent_date_modified_for_events is off by default. (too avoid confusion with Date Modified not matching Windows Explorer)
The date modified information will be lost when the index is rebuilt.

– It never states that "FILETIME" format is acceptable for searches too. For example:

This is a new feature in Everything 1.5 and I haven't updated the documentation for ES yet.

Only FILETIMES > 99999999 are accepted. (to avoid conflicts with ISO 8601 YYYYMMDD)

Here comes another caveat: format accepts either dm:^>X or dm:>"X", but not "dm:>X"

Search function will not work when the function name is quoted, even from ES.
You can use quotes anywhere after the colon (:)

For example:
dm:">132851492644565250"
dm:">"132851492644565250

(Did these not work for you?)

I'll make a note in the ES documentation: Alternatively, use double quotes to escape \ & | > < ^

[aleksusklim]

update_parent_date_modified_for_events is off by default.

I understood, I just said that I decided to design my program to work either way, with or without that option (and it won't hurt even if enabled).

But I can see how this behavior could be useful in other cases.

This is a new feature in Everything 1.5 and I haven't updated the documentation for ES yet.

Oh, I see. It's very convenient in automated processing: take the date from a file and the directly compare with it.

Search function will not work when the function name is quoted, even from ES.

Hm-m… I thought I'm quoting the command-line argument and not a search string!

Now I understand that ES.exe is designed to parse rightmost command-line parameters as RAW query string, not as separated arguments.

(Did these not work for you?)

That works, I just have a habit of quoting any non-constant (that are taken from user input) parameters when invoking other processes ;)
But now I see that all quotes are parsed as literal quotes, so even "-csv" in quotes isn't working.

I'll make a note in the ES documentation

Suddenly, I have one more question:

Code: Select all

es -instance 1.5a -clear-settings
Settings saved.

es -instance 1.5a -csv -no-header -csv -dm "C:\Test\" -n 1
28/12/2021 12:19,"C:\Test\Folder_1"

es -instance 1.5a -csv -size -save-settings
Settings saved.

es -instance 1.5a -csv -no-header -csv -dm "C:\Test\" -n 1
16998,28/12/2021 12:19,"C:\Test\Folder_1"

– So, basically, if someone will invoke es.exe with -save-settings then my program effectively breaks, because Everything will mix stored settings with current parameters of my query.

I cannot see any options in ES.exe that will make it "not load stored settings". I can see only:

Switches can be disabled by prefixing them with no-, eg: -no-size.

But this will require me to prefix every single unused switch, won't it?
In other software I saw options like --no-config for wget.exe or /D for cmd.exe to disable reading of stored configuration.

Does ES.exe have something similar?

[void]

But this will require me to prefix every single unused switch, won't it?
In other software I saw options like --no-config for wget.exe or /D for cmd.exe to disable reading of stored configuration.

Does ES.exe have something similar?

I have put on my TODO list to add a -no-config command line option.
A little tricky to do at the moment as ES loads the config before parsing the command line arguments.

Thanks for the suggestion.

[void]

Everything 1.5.0.1291a replaces update_parent_date_modified_for_events with update_parent_date_modified_for_all_events.