How to find similar videos? Does properties help?

[Jerry2024]

Now I use Everything to find duplicate videos by
1) same file size (exactlly same video file)
2) same video length (compressed video file / videos added watermark)

However, if the video is edited by adding 3-second advertisement in front of original video content, the file size / video length both change, and above ways do not work any more. I call them "similar videos", but actually they are the same excep advertisement.

The version I use is 1.5 alpha, and it has "Property" index, search and duplicate file finding function, so does this help to find similar videos?

Thanks!

[void]

Everything probably wont help here..



If you're only after duplicates of one video, you could search for the length + 3 seconds.

For example:

If the original length is: 5:50

Search for:

length:5:53



To find media files that are within 3 seconds:
video: addcol:a a:=$length:/30000000 dupe:a-descending

The new column A will show the duration in seconds / 3.
This doesn't quite work because duplicated files may fall into different 3-second brackets.
It might be useful enough to give you some duplicated results..

$length: is replaced with the raw length in 100nanoseconds.
Try dividing by larger numbers for less brackets, for example:
video: addcol:a a:=$length:/100000000 dupe:a-descending
(10 second brackets)

[ChrisGreaves]

However, if the video is edited by adding 3-second advertisement in front of original video content, the file size / video length both change, ...

Hi Jerry. I suspect that there is little difference between the handling of audio files and video files. I mentioned some of the problems with audio files in How to find many names (partial match of different names) of music files on disk?
The problem is soluble in Audio, or at least, in MP3 files with its packets of data, but may not be soluble that way in Video. I know little of video formats.
Cheers, Chris

[therube]

I use hashmedia.bat to find same "content" (for as far as I understand it) between two otherwise video files that do not size/hash compare.

As in, the video "streams" may be the same, where the video files are not.


1) can be done by checking the file hash (which Everything 1.5 alpha can do, like with a md5 hash Property).
(Same can be done by selecting various files in Everything & SendTo some sort of hash program.)

2) not certain, but i believe the watermark physically changes the video "stream" itself would not compare.
video Lengths Property can again be handled by Everything 1.5 alpha.

[Jerry2024]

Everything probably wont help here..

If you're only after duplicates of one video, you could search for the length + 3 seconds.
For example:
If the original length is: 5:50
Search for:
length:5:53

Thanks, I search for more than 1000 video files per time, so I do not know the orginal length of videos. Actually I want to find all duplicate videos on my disk and remove them. I tried some software like Gemini & Duplicate File Finder, but they just find the precisely same files (by size or hash).

The length property helps a lot to find duplicated (differnt size) videos, so hope it maybe enhanced in the future. Will it be considerd to take similar length video file as duplicate file and show in duplicate file list, and allow user to set how many seconds to ignore ? For example, if user set ""ignore 5 seconds difference", the following files will be take as duplicated files, and users may check mannually by video preview picture.
A video: 50:30 / B video: 50:35
C video: 3:20 / D video: 3:22
E video: 1:58:10 / F video: 1:58:12 / G video: 1:58:15

[Jerry2024]

However, if the video is edited by adding 3-second advertisement in front of original video content, the file size / video length both change, ...

Hi Jerry. I suspect that there is little difference between the handling of audio files and video files. I mentioned some of the problems with audio files in How to find many names (partial match of different names) of music files on disk?

Thanks Chris, I will read and study

[Jerry2024]

I use hashmedia.bat to find same "content" (for as far as I understand it) between two otherwise video files that do not size/hash compare.

Thanks therube, but it seems a little hard for me, but I will try to understand

[therube]

similar length video file as duplicate file and show in duplicate file list, and allow user to set how many seconds to ignore ?

Tools | Options | Advanced -> length_dupe_tolerance=5

Oh, & WinMerge can also compare videos.
Not sure just what methodology it uses, but you can throw pairs of videos to it & see what it gives (or not).
(Theoretically you'd throw what you feel are identical videos to it, but not hash identical.)

[santilli]

As pointed, for this particular use case Everything will not suffice. It's great for spotting duplicates based on name, duration, file size and checksum. If you're looking for duplicates that have diverging properties, it's best to use a tool made specifically for spotting video duplicates. I use Czkawka as it is super simple and easy to use, and you can set a similarity threshold to streamline the results, that helps spot files that goes from 100% duplicate, to extremely similar, down to just slightly similar (false positives).

[Jerry2024]

similar length video file as duplicate file and show in duplicate file list, and allow user to set how many seconds to ignore ?

Tools | Options | Advanced -> length_dupe_tolerance=5

Oh, & WinMerge can also compare videos.
Not sure just what methodology it uses, but you can throw pairs of videos to it & see what it gives (or not).
(Theoretically you'd throw what you feel are identical videos to it, but not hash identical.)

Thanks! I'll try it.

[Jerry2024]

As pointed, for this particular use case Everything will not suffice. It's great for spotting duplicates based on name, duration, file size and checksum. If you're looking for duplicates that have diverging properties, it's best to use a tool made specifically for spotting video duplicates. I use Czkawka as it is super simple and easy to use, and you can set a similarity threshold to streamline the results, that helps spot files that goes from 100% duplicate, to extremely similar, down to just slightly similar (false positives).

OK I'll try Czkawka, sounds great, thanks!

[Jerry2024]

similar length video file as duplicate file and show in duplicate file list, and allow user to set how many seconds to ignore ?

Tools | Options | Advanced -> length_dupe_tolerance=5

Oh, & WinMerge can also compare videos.
Not sure just what methodology it uses, but you can throw pairs of videos to it & see what it gives (or not).
(Theoretically you'd throw what you feel are identical videos to it, but not hash identical.)

While I also noticed you provide "Tools | Options | Advanced -> length_dupe_tolerance=5" solution, thanks again!

[Jerry2024]

similar length video file as duplicate file and show in duplicate file list, and allow user to set how many seconds to ignore ?

Tools | Options | Advanced -> length_dupe_tolerance=5

Oh, & WinMerge can also compare videos.
Not sure just what methodology it uses, but you can throw pairs of videos to it & see what it gives (or not).
(Theoretically you'd throw what you feel are identical videos to it, but not hash identical.)

Length dupe olerance works now! Thanks!!

Additionally it should be 5000 not 5 ^^

[therube]

1. file sizes are different, so it fails a hash comparison
2. if you compare the "properties" of the files, they're essentially the same, just some ancillary differences
3. WinMerge shows the files to be different
from the look of it, the difference is in the last "line" of the file (all else being the same, identical)
the last "line" (in this case), can be thought of as the files "meta-data"
4. a different comparison program, gives results like 3, where the differences are towards the end of the file
& again, presumably "meta-data"
5. comparing the data portions of the files, you can see both the video & audio data are identical
.

hashvideo-files-do-compare.png (295.55 KiB) Viewed 2668 times

[therube]

1. comparing the file properties, all that data is identical, only 1 specifies the language, the other, not
2. comparison of the data portions of the clips shows that the audio is identical, but the video is not
3. visual comparison, points out that 1 clip has a watermark (not too clear in the shot, but it is there)
& the other does not
.

Everything - hashvideo 2.png (93.89 KiB) Viewed 2666 times

[therube]

Czkawka

I'd been meaning to look at that for some time now, but hadn't gotten around to it (till now).

(with a quick, very quick, look & without reading any docs)

(for me) leaves much to be desired
deals with directories, directory trees, so that is a great failing, as your files need to be within particular directories
(about the only thing that doesn't deal with directories, is Everything)

the "Similarity" slider appears to filter what is displayed (or not)
once you have the list of "dups", other then that Cz has pointed them out, not really much of anything you can do with that knowledge - except open said files in your video player

(& it cannot open multiple files, simultaneously, so you have to open 1, then another.
Everything can open multiple files simultaneously .)

actual results. hit or miss.
on clips that were similar in "concept" but totally unrelated in content, it said "ah, concept is the same, so they must be the same", but they were not
on clips that were the same "content", it fared a bit better

[therube]

Oh, on hashmedia.bat.

There are times when the "specs" on a pair of files is identical, where viewing the files looks identical, but where the physical hash of the files data is not identical, but in the end, to ones (weary) eyes, they're the same.

[Jerry2024]

Czkawka

I also tested Czkawka, but finally back to Everything, since it does not support big preview thumbnail and it's not convenient to open the file path.

Fow now I use the following way to find similar videos, though mayb not perfect, but it still helps me a lot.
1) Index property "length" in Everything;
2) Set length tolerance "2000" (2 second) in Everything option;
3) Search video files , rank by length, and choose "Find Length Duplicate";
4) View the search result list by "View-Large thumbnails", or extra large;
5) Compare videos by preview thumbnail / manually, and usually if the video is similar (not same size, and only within 2 seond length difference), they will show nearby. If thumbnail doesn't show useful frame, I'll open the file.

Also hope this summary may help guys who is finding similar video duplicate sulotion.