Special characters not found in eml files (content search)?

[Biff]

It seems as if the content search (content:") (sometimes) would find in *.eml files / e-mails (stored outside of Thunderbird and inside) only a part of the text or maybe would not find special charakters. It perhaps would not find charakters like ä, ü, ß, /, spaces in the text in the "Subject" field and in the body of the e-mail / eml file...trying again something of that now is found.

This is not found here (the entire line with colon) in an eml file (in others it is or might be):
: 0,0 kWh
in search field:
content:": 0,0 kWh

This is found:
Nummern:

This is found:
5 - M22071 |
DE-14-MI001-PTB006


This is not found:
5 - M22071 | DE-14-MI001-PTB006
071 | DE
1 | D

Found:
1 |

How could one (always) find such?

[tuska]

With this BOOKMARK I was able to carry out a successful search:

1. Code: Select all

   Name: Content\Mytext: https://www.voidtools.com/forum/viewtopic.php?p=33633#p33633 mycontent:mytext: | ansicontent:mytext: | utf8content:mytext: | utf16content:mytext: |  unicodecontent:mytext: | ifilter-content:mytext: | mc:SearchWord | https://www.voidtools.com/forum/viewtopic.php?t=13506

2. "Match case" bis inclusive "Match path" und Regular expressions: Disabled
3. "Match prefix" bis inclusive "Ignore whitespace": (No change)
4. Search: Custom...

   Code: Select all

   content:mytext: | plain-text-content:mytext: | unicodecontent:mytext: | ifilter-content:mytext:

5. Filter: Everything
6. "Columns" bis inclusive "Index:": (No change)
7. Macro:

   Code: Select all

   mc<mytext>

   I.e. mc:"SearchTerm" executes the command "content:mytext: | plain-text-content:mytext: | ..."!
    
8. Source - NotNull & Important information | Q: Creating a shortcut for "content:"?

 
With these search queries you should find what you are looking for:

Code: Select all

ext:eml mc:"ä, ü, ß,"
ext:eml mc:": 0,0 kWh"
ext:eml mc:"5 - M22071 | DE-14-MI001-PTB006"
ext:eml mc:"071 | DE"
ext:eml mc:"1 | D"

ext:eml mc:"5 - M22071 | DE-14-MI001-PTB006"|"071 | DE"|"1 | D"

[Biff]

Many thanks!

So Everything obviously really doesn't find such by default.

1-7 done:

With these search queries you should find what you are looking for:

Combined with using that bookmark? I have insert one of the lines to the search? So when I search for one of these characters I have to insert one of the corresponding lines of the code in the search field?

There is no way to ALWAYS find these characters (without having to add additional lines) to the search field?

Thank you very much again.

When clicking the bookmark this is shown:


And then for example:


That is correct?

The icon is not shown I chose for the bookmark:


Something seems to have gone wrong during the search. Everything is greyed out, no access / respond anymore.

[tuska]

I don't think I understand your answer.

Instead of content:"searchword" just enter mc:"searchword" in the Everything search box
and you will find your text content for .eml files!

The search queries - as they should be in the Everything search box - are shown above.

[Biff]

I don't think I understand your answer.

And even the questions? That's a pity, very sorry about that! Why is that? What exactly don't you understand?

Instead of content:"searchword" just enter mc:"searchword" in the Everything search box
and you will find your text content for .eml files!

Instead of content:"searchword" just enter mc:"searchword" in the Everything search box
and you will find your text content for .eml files!

Oops, if it's that easy, then what's the bookmark and everything else for? Obviously we both do not understand the answers (and perhaps the questions) of each others.

Very sorry again.

[tuska]

2Biff
Simply enter the search queries in the search box as I have shown above.
In this case, the commands from the bookmark will be used.

However, do not call up the bookmark via the menu.
In practice, you should then get the desired results.

Unfortunately, I cannot contribute any more to this topic.

I do not know about any further technical background.
If you need more information, you may find it here.
The topic has already been linked above under point 8.

[Biff]

Simply enter the search queries in the search box as I have shown above.
In this case, the commands from the bookmark will be used.

OK, sorry for my confusion. These queries only work with that bookmark (or they do not). Obviously I am using them wrong, nothing shown (besides of the green progress bar in the status bar for some time, then it disappears.

However, do not call up the bookmark via the menu.

Oops, OK, so the bookmark may not be used as a bookmark. If I had known that...

In practice, you should then get the desired results.

OK, somehow the practice with me obviously is another one. A practice of confusion or so. Nothing found here.

Unfortunately, I cannot contribute any more to this topic.
I do not know about any further technical background.

OK, understandable, that was a lot of background. So nothing more I could misunderstand.

Thank you very much!

[tuska]

2Biff
You could try one more thing.

Change in the bookmark the content of Search: Custom ... to these values:

Code: Select all

content:mytext: | ansicontent:mytext: | ifilter-content:mytext: | plain-text-content:mytext: | unicodecontent:mytext: | utf8content:mytext: | utf16content:mytext:

Then perform a restart of Everything: /restart <enter>

Carry out your search queries now, e.g.

Code: Select all

ext:eml mc:"071 | DE"

Remark:
With this setting, however, I also had unexpected or no search hits...
However, the search queries announced above work for me without any problems.

Unfortunately, I can't contribute any more to this topic,
as I only have a few test files (.eml) and use Outlook.

[Biff]

Many thanks!

Cannot get it working here with the best will in the world.

OK, I then just will search for other characters.

Unfortunately, I can't contribute any more to this topic,
as I only have a few test files (.eml) and use Outlook.

Yes, I understand, many thanks for your great help!

[void]

eml files are usually stored as text/html
They can sometimes be stored as text/plain

Both are supported.



There's probably some weird html character escaping.

Please try the following search:

addcolumn:regmatch1 ext:eml regex:content:"(5 - M22071.*)"

The content Everything is reading will be shown in the new Regular expression match 1 column.

[tuska]

2void

Please try the following search:

Code: Select all

addcolumn:regmatch1 ext:eml regex:content:"(5 - M22071.*)"

With this search query, only the column is displayed, but I do not get any search results.
But maybe that's different for user Biff?

Remark:
... I also had unexpected ... search hits...

I carried out the test as follows.

• Text content of an .eml file in ANSI format (Total Commander: Encoding: ANSI)
  1. Notepad++ shows this text in an .eml file before a text block: Content-Type: text/plain; charset="UTF-8"
  2. German umlauts ä, Ä, ö, Ö, ü, Ü, and special character 'ß' ... inserted in the text block with Notepad++
  3. Further texts - as announced in the forum - inserted with Notedpad++ in an .eml file

For German umlauts and special character, Everything finds two .eml files (of three) that do NOT contain these characters.
The file content was viewed with EmEditor Professional (64-bit) version 23.1.0 from 24.1.2024.

Remark:
For comparison, an expected search result was obtained in Total Commander,
i.e. only one .eml file was found instead of a total of three .eml files.

I therefore suspect that some code stored in the bookmark cannot handle the German umlauts and the special character.
But that could still be tested out which code this is exactly.

2Biff
Have you saved *.eml in Everything options - 1.5.0.1366a mod1 (1367a) (x64) > Content > [x] Index file content > Include only files: ?
If not, please add this file extension and see if you get a search result.

But perhaps this query will also help?
For example notindexed::

Code: Select all

ext:eml notindexed:mc:"Searchword"

Bookmark - Makro: mc<mytext>
I am no longer sure whether the macro is case-sensitive.
To be on the safe side, in this case I recommend performing a search query with the macro in lower case: mc:"Searchword".

I have made an addition to the code for the bookmark (utf16becontent:mytext:) ...
change in the bookmark the content of Search: Custom ... to these values:

Code: Select all

content:mytext: | ansicontent:mytext: | ifilter-content:mytext: | plain-text-content:mytext: | unicodecontent:mytext: | utf8content:mytext: | utf16content:mytext: | utf16becontent:mytext:

[void]

Currently, Everything only supports the UTF-8 charset.

I'll look into support for other charsets.

[Biff]

eml files are usually stored as text/html
They can sometimes be stored as text/plain

OK, so they are saved with the extension eml with the character sets of text/plain/html (or so).

addcolumn:regmatch1 ext:eml regex:content:"(5 - M22071.*)"

This ("5 - M22071") is found with
content:
as well. With
content:"
additionally are found the other files with other extensions. The CPU goes up to about 90 % (by using "content:") when I search ALL drives / everything.

addcolumn:regmatch1 ext:eml regex:content:"(5 - M22071.*)"
finds two eml files.

This
5 - M22071 | DE-14-MI001-PTB006
was not found before with
content:"
Now I find it with
content:"

Maybe I didn't note the space here (" 5") before, don't know:
content:" 5 - M22071 | DE-14-MI001-PTB006
This finds nothing.

This finds 7 txt files (without the space (" 5")):
content:"5 - M22071 | DE-14-MI001-PTB006


This finds two eml files (which were not find with: "content:"5 - M22071 | DE-14-MI001-PTB006") even with a space before the "5":
addcolumn:regmatch1 ext:eml regex:content:"( 5 - M22071 | DE-14-MI001-PTB006.*)"

The content Everything is reading will be shown in the new Regular expression match 1 column.

Sorry, I do not understand, match 1 column? What column? New Regular expression?

Have you saved *.eml in Everything options - 1.5.0.1366a mod1 (1367a) (x64) > Content > [x] Index file content > Include only files: ?

This is how it is here:

But perhaps this query will also help?
For example notindexed::

Thank you for the link. Yes, maybe it will, but I am afraid that is too complicated for me.

Such a seemingly simple search seems to be extremely complicated (much too complicated for a simple worker), as I now realize. Just to remember all the options / functions (for a person not having any idea of nothing) to use for a search the next time when I search for text (I even had to remember / know that I had to search for special characters and use such options at all) in everyday life is much too hard, I guess.

I have made an addition to the code for the bookmark (utf16becontent:mytext:) ...

Ah, great, thank you very much!

[tuska]

Sorry, I do not understand, match 1 column? What column? ...

2024-01-27_addcolumn-regmatch1.png (9.2 KiB) Viewed 5596 times

⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺

But perhaps this query will also help?
For example notindexed::

Thank you for the link. Yes, maybe it will, but I am afraid that is too complicated for me.

2024-01-27_Extension EML Notindexed.png (19.05 KiB) Viewed 5596 times

[Biff]

Ah, yes, now I see, thank you very much for the screenshots, I like them very much!

[Thy Grand Voidinesss]

[...]
I have made an addition to the code for the bookmark (utf16becontent:mytext:) ...
change in the bookmark the content of Search: Custom ... to these values:

Code: Select all

content:mytext: | ansicontent:mytext: | ifilter-content:mytext: | plain-text-content:mytext: | unicodecontent:mytext: | utf8content:mytext: | utf16content:mytext: | utf16becontent:mytext:

If I understood correctly this discussion- using a Bookmark with the above code will make Everything look for literally

mytext

while making sure that different coding pages will be taken into account and thus assuring that some weird / unusual cases will also be taken into consideration?


Is there a way to rework this Bookmark so that there would be just one such

Code: Select all

mytext

place in its code [preferably at its very end] so that a user could just leave it empty- for the purpose of writing later on / on the spot what to look for after choosing [i.e. activating] said Bookmark


Or maybe a Filter equivalent of it that can be created- which would have an extra benefit of initially having Search Edit box empty thus not cluttered with a code?

[void]

Everything 1.5.0.1367a improves eml content searching.

The following charsets are now supported:

• ansi_x3.4-1968
• ascii
• asmo-708
• big5
• cp1025
• cp1252
• cp866
• cp875
• csiso2022jp
• dos-720
• dos-862
• euc-cn
• euc-jp
• euc-kr
• gb18030
• gb2312
• gbk
• hz-gb-2312
• ibm-thai
• ibm00858
• ibm00924
• ibm01047
• ibm01140
• ibm01141
• ibm01142
• ibm01143
• ibm01144
• ibm01145
• ibm01146
• ibm01147
• ibm01148
• ibm01149
• ibm037
• ibm1026
• ibm273
• ibm277
• ibm278
• ibm280
• ibm284
• ibm285
• ibm290
• ibm297
• ibm420
• ibm423
• ibm424
• ibm437
• ibm500
• ibm737
• ibm775
• ibm850
• ibm852
• ibm855
• ibm857
• ibm860
• ibm861
• ibm863
• ibm864
• ibm865
• ibm869
• ibm870
• ibm871
• ibm880
• ibm905
• iso-2022-jp
• iso-2022-kr
• iso-8859-1
• iso-8859-13
• iso-8859-15
• iso-8859-2
• iso-8859-3
• iso-8859-4
• iso-8859-5
• iso-8859-6
• iso-8859-7
• iso-8859-8
• iso-8859-8-i
• iso-8859-9
• johab
• koi8-r
• koi8-u
• ks_c_5601-1987
• macintosh
• shift_jis
• unicodefffe
• us-ascii
• utf-16
• utf-32
• utf-32be
• utf-7
• utf-8
• windows-1250
• windows-1251
• windows-1252
• windows-1253
• windows-1254
• windows-1255
• windows-1256
• windows-1257
• windows-1258
• windows-874
• x-chinese_cns
• x-cp20001
• x-cp20003
• x-cp20004
• x-cp20005
• x-cp20261
• x-cp20269
• x-cp20936
• x-cp20949
• x-cp50227
• x-ebcdic-koreanextended
• x-europa
• x-ia5
• x-ia5-german
• x-ia5-norwegian
• x-ia5-swedish
• x-iscii-as
• x-iscii-be
• x-iscii-de
• x-iscii-gu
• x-iscii-ka
• x-iscii-ma
• x-iscii-or
• x-iscii-pa
• x-iscii-ta
• x-iscii-te
• x-mac-arabic
• x-mac-ce
• x-mac-chinesesimp
• x-mac-chinesetrad
• x-mac-croatian
• x-mac-cyrillic
• x-mac-greek
• x-mac-hebrew
• x-mac-icelandic
• x-mac-japanese
• x-mac-korean
• x-mac-romanian
• x-mac-thai
• x-mac-turkish
• x-mac-ukrainian
• x_chinese-eten

text/html => text/plain has been improved.
text/html => text/plain (if it exists) is now used over text/plain.



Just a reminder: The system eml iFilter is disabled by default in Everything as the stock Windows eml iFilter modifies eml files when reading content.

If you wanted to install a third party eml iFilter you will need to disable the built in Everything eml content handler:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  eml
• Select: content_builtin_eml_handler
• Set the value to: false
• Click OK.

Do not disable content_builtin_eml_handler if you are using the stock Windows eml iFilter.



The system eml property handler is disabled by default in Everything as the stock Windows eml property handler modifies eml files when reading properties.

If you wanted to install a third party eml property handler you will need to enable eml property handlers:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  eml
• Select: property_system_eml
• Set the value to: true
• Click OK.

Do not enable property_system_eml if you are using the stock Windows eml property handler.

content:mytext: | ansicontent:mytext: | ifilter-content:mytext: | plain-text-content:mytext: | unicodecontent:mytext: | utf8content:mytext: | utf16content:mytext: | utf16becontent:mytext:

Please don't use this filter.
eml files are not text/plain.

[tuska]

content:mytext: | ansicontent:mytext: | ifilter-content:mytext: | plain-text-content:mytext: |
unicodecontent:mytext: | utf8content:mytext: | utf16content:mytext: | utf16becontent:mytext:

Please don't use this filter.
eml files are not text/plain.

I have a few test files with file extension .EML with encoding "ANSI"
and the search for file contents works for me without any problems.

Maybe ansicontent: has helped here.

I have defined the bookmark with operator "OR" as I am never sure or would have to check beforehand
which encoding a file has when I want to search for file contents.


2Thy Grand Voidinesss
I'll get back to you later...

[horst.epp]

My Thunderbird MailDir files are .eml encoded as UTF-8 without BOM.
Everything indexed content search finds all German Umlauts without problems.

The EML related settings are:
content_builtin_eml_handler=1
property_system_eml=0

Also, the OS delivered .EML "Microsoft Windows Mail Mime Preview Handler" works in Everything.

[tuska]

If I understood correctly this discussion- using a Bookmark with the above code will make Everything look for literally mytext
while making sure that different coding pages will be taken into account and thus assuring that some weird / unusual cases
will also be taken into consideration?

Yes, as long as the encodings are supported by Everything.
I have taken the liberty of deleting the text that I do not think is applicable.

Is there a way to rework this Bookmark so that there would be just one such

Code: Select all

mytext

place in its code [preferably at its very end] so that a user could just leave it empty- for the purpose of writing later on / on the spot
what to look for after choosing [i.e. activating] said Bookmark

Or maybe a Filter equivalent of it that can be created- which would have an extra benefit of
initially having Search Edit box empty thus not cluttered with a code?

For me, the file content search mainly makes sense with a macro, i.e. mc: (mycontent:), content:; si: (Windows Search)
or whatever macro abbreviation you want to use.

I do not find it necessary to revise the bookmark, as it is usually only set up once
and only in exceptional cases a small adjustment might be necessary afterwards.

However, a file content search with a filter is also possible - here are a few additional examples:
- Search content automatically without "content:" modifier?
- Is it possible without "tags:xxxx" to search tags field data ??
- Preset / favor Windows Search Index

Every user as he likes [or the circumstances require it].

[void]

Just a small note:

Everything 1.5.0.1368a adds a eml-content: search function.

eml-content: will treat file content as message/rfc822.

This will allow you access to Everything's eml content search if you use disable Everything as the default eml handler.
This will also allow you to search eml content that is using a different extension.



Usage:

*.eml eml-content:"foo bar"
*.eml.backup eml-content:"foo bar"

[Biff]

Many thanks!

This will allow you access to Everything's eml content search if you use disable Everything as the default eml handler.

If one does not use Everything as the default eml handler you don't need it? Where can I find the option (or whatever it is) Everything as the default eml handler? So then why should one not use Everything as the default eml handler?

This will also allow you to search eml content that is using a different extension.

xamples:
"eml-content:"abc 123"

So maybe this would search in every file / folder set in "Content" tab (very slowly)? But this would not search eml messages stored in a special Thunderbrid file (that Thunderbird uses to store e-mails)?

Respectively how would one search for both at the same time, content like usual and content of e-mails?

[void]

If one does not use Everything as the default eml handler you don't need it?

No, it's not required.

With the Everything eml handler enabled, the following searches are the same:

*.eml content:"abc 123"
*.eml eml-content:"abc 123"

Where can I find the option (or whatever it is) Everything as the default eml handler?

Tools -> Options -> Advanced -> content_builtin_eml_handler
Please don't disable this if you are using the stock Windows eml handler.
The stock Windows eml handler will modify your eml files.
This is the main reason I added a built-in eml handler.

So then why should one not use Everything as the default eml handler?

No reason to ever disable the built-in Everything eml handler unless you want certain functionality from a third party eml iFilter.
Do not disable the Everything eml content handler if you are not using a custom third party eml iFilter.

So maybe this would search in every file / folder set in "Content" tab (very slowly)?

Yes, eml-content: will search all files/folders.
eml-content: ignores the extension.

Ideally you would specify the extension with your eml-content: search.

For example:

*.eml eml-content:"abc 123"
*.eml.backup eml-content:"abc 123"

But this would not search eml messages stored in a special Thunderbrid file (that Thunderbird uses to store e-mails)?

I believe Thunderbrid uses message/rfc822 encoding for its eml files so it should work.

Respectively how would one search for both at the same time, content like usual and content of e-mails?

Just use the content: search function.
The content: search function will work out what iFilter to use based on the file extension.
eml-content: just gives you more control by allowing you to search other file extensions.

[Biff]

OK, I do not use any external filter or whatever with Everything.

The stock Windows eml handler will modify your eml files.

Oops, sounds dangerous, how? And why? I guess, it would not be good, if that happened.

Tools -> Options -> Advanced -> content_builtin_eml_handler
Please don't disable this if you are using the stock Windows eml handler.

I don't (intentionally). How could one find out? So this (content_builtin_eml_handler) is activated by default obviously. And it should be left like it is here, I guess.

I believe Thunderbrid uses message/rfc822 encoding for its eml files so it should work.

I meant these folders or files in which Thunderbrid obviously stores the e-mails:

Yes, eml-content: will search all files/folders.
eml-content: ignores the extension.

OK, so there is / might be eml-content in files others than e-mails / eml files, hadn't expected that (how could one know what for to search, eml-content or usual then)? The content search finds eml-content (but not other content) in every file (defined in the Content tab), mainly including e-mails / eml files.

Respectively how would one search for both at the same time, content like usual and content of e-mails?

Just use the content: search function.

So the content search now finds the content of eml files as well? Can they can be found with the normal AND the eml content search?
So one does not need searches like these

*.eml eml-content:"abc 123"
*.eml.backup eml-content:"abc 123"

to find content of e-mails, but with these ones one can search for e-mails with eml extension / eml.backup extension only, for example?

Very sorry for my confusion.

So finally (perhaps) I even do not need to do / change anything? Just go on using the old content search for everything?

[void]

Oops, sounds dangerous, how? And why? I guess, it would not be good, if that happened.

You will have to ask Microsoft why their eml iFilter modifies eml files.

I don't (intentionally). How could one find out?

ShellExView can be used to manage iFilters.

I meant these folders or files in which Thunderbrid obviously stores the e-mails:

No, Everything doesn't have support for msf files.
I think Thunderbird has an option to store emails as message/rfc822 eml files.

OK, so there is / might be eml-content in files others than e-mails / eml files, hadn't expected that (how could one know what for to search, eml-content or usual then)?

message/rfc822 are generally stored in eml files.
There's no file signature for message/rfc822 files.
You will need to know which extensions contain message/rfc822.

Please use the following to search the content of eml files:

*.eml content:"abc 123"

So the content search now finds the content of eml files as well?

It always has.
content: will use the extension associated iFilter to search content.
For eml files, content: will use the built-in eml content handler.

Can they can be found with the normal AND the eml content search?

No need to use content: AND emlcontent:
Just use content:

[Biff]

You will have to ask Microsoft why their eml iFilter modifies eml files.

OK, then I might as well ask them why they don't make a decent operating system.

ShellExView can be used to manage iFilters.

Many thanks for the link!

Please use the following to search the content of eml files:

*.eml content:"abc 123"

So this - in a bookmark - is OK?
/replace-sel #nbsp:.eml content:"

So the content search now finds the content of eml files as well?

It always has.

Sorry, but I thought, based on what this thread is about, apparently not. And I thought that was the reason why there was now this new search function for eml files / e-mails, but apparently that's not the case at all. So if you can find all content with content:, what for is the content search especially for eml / e-mails? Very sorry for my confusion again.

Can they can be found with the normal AND the eml content search?

No need to use content: AND emlcontent:
Just use content:

So for EACH file extension, including eml. So then for me, when I search, EVERYTHING remains as it was. Holy...unbelievable, I started a thread, which I was very proud of, and obviously I absolutely didn't understand what about the entire thread is.

[void]

So this - in a bookmark - is OK?
/replace-sel #nbsp:.eml content:"

Yes, that looks fine.
That will search your eml files for the specified content.

Sorry, but I thought, based on what this thread is about, apparently not. And I thought that was the reason why there was now this new search function for eml files / e-mails, but apparently that's not the case at all. So if you can find all content with content:, what for is the content search especially for eml / e-mails? Very sorry for my confusion again.

Just use content: and ignore emlcontent:

content: uses the extension associated iFilter. (this will be the built-in eml iFilter for eml files by default)
emlcontent: ignores the extension and will perform a content search with the built-in eml iFilter.

[Biff]

es, that looks fine.
That will search your eml files for the specified content.

OK, thank you.

Just use content: and ignore emlcontent:

content: uses the extension associated iFilter. (this will be the built-in eml iFilter for eml files by default)
emlcontent: ignores the extension and will perform a content search with the built-in eml iFilter.

OK, many thanks, I will do so. So just like I did before (now with some confusion).

I think I better should stop reading and asking in my threads, then everything will be a lot easier.

[void]

Everything 1.5.0.1374a adds encoding support for the following message/rfc822 (eml) properties:

• From
• Subject
• To

[Biff]

Thank you very much!