Remove unreliable duplicate-lesson-file matching

Filename-similarity matching within a folder can't distinguish a real
re-download duplicate from a course that splits one topic across
several numbered files ("...Part 1"/"...Part 2", "-1"/"-2"/"-3") -
both look like near-100% matches by name alone, and the latter is a
common, completely normal pattern. This produced hundreds of false
positives in practice. Removes Duplicate Lesson Files entirely
(detection function, its two API routes, and the File Management UI
section); whole-course-folder matching (Duplicate Courses) is
unaffected and stays the only duplicate scan in the app.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -1,3 +1,4 @@
+2026-08-25 12:25 UTC — Remove unreliable duplicate-lesson-file matching
 2026-08-25 00:33 UTC — Widen thumbnail sampling window past platform bumpers
 2026-08-25 00:26 UTC — Fix race condition in thumbnail candidate generation
 2026-08-25 00:20 UTC — Add autoplay, progress rings, duplicate-lesson detection, title-card thumbnails

README.md

@@ -185,20 +185,17 @@ silently stay blank instead of erroring.
   difference doesn't hide a real duplicate. Groups by similarity (union of
   any two courses over the threshold, transitively) rather than showing
   raw pairs; each course in a group gets a one-click Hide, or a **Delete**
-  that permanently removes it from disk (confirmed with the full path
+  that permanently removes it from disk (the one destructive action in the
-  before it runs). A group can also be marked **"Not a duplicate"** if the
+  whole app - confirmed with the full path before it runs). A group can
-  match is wrong, which excludes that specific pair from future scans
+  also be marked **"Not a duplicate"** if the match is wrong, which
-  without touching anything else that happens to match one of those
+  excludes that specific pair from future scans without touching anything
-  courses; ignored pairs are listed (and reversible) under "Ignored
+  else that happens to match one of those courses; ignored pairs are
-  matches," and persist through backup/restore alongside hidden paths and
+  listed (and reversible) under "Ignored matches," and persist through
-  Next Up.
+  backup/restore alongside hidden paths and Next Up. (Matching is
-  - *Duplicate Lesson Files*: the same idea one level down - media files
+  deliberately scoped to whole course folders, not individual lesson
-  sitting in the same folder that look like the same lesson downloaded
+  files - courses that split one topic across several numbered files,
-  twice (e.g. a re-download that landed alongside the original instead of
+  e.g. "...Part 1"/"...Part 2", look just as similar by name as a genuine
-  replacing it). Compares only within a folder, at a higher match
+  re-download, so file-level matching wasn't reliable enough to keep.)
-  threshold than Duplicate Courses, so two different lessons on a similar
-  topic don't get flagged; shares its ignore list with Duplicate Courses.
-  Delete here removes a single file, not a whole course.
   - *Clean Up Stale References*: finds entries in the hidden-paths list,
   Next Up queue, Favorites, or Recently Viewed history that point at a path
   no longer on disk - normally from renaming/moving/deleting a course

VERSION

@@ -1 +1 @@
-2026-08-25 00:33 UTC — widen thumbnail sampling window past platform bumpers
+2026-08-25 12:25 UTC — remove unreliable duplicate-lesson-file matching

offlineu_core.py

@@ -1864,64 +1864,6 @@ def _find_duplicate_courses(min_similarity: float = 0.6) -> List[Dict[str, Any]]
     return groups
 
 
-def _find_duplicate_lesson_files(min_similarity: float = 0.75) -> List[Dict[str, Any]]:
-    """
-    Within each course, media files sitting in the same folder whose
-    names look like the same lesson downloaded twice - e.g. "03 Setting
-    Up Your Environment.mp4" and "03 Setting Up Your Environment (1).mp4"
-    left behind by a re-download that landed in the same section. Scoped
-    to siblings in the same folder, unlike Duplicate Courses' whole-library
-    comparison: two genuinely different lessons can share a lot of
-    vocabulary ("Part 1"/"Part 2" of a topic) without being duplicates, so
-    same-folder pairing plus a higher similarity threshold keeps this
-    conservative. Shares Duplicate Courses' ignore list (get_ignored_
-    duplicate_pairs) - a pair is just a pair of paths there, whether
-    they're course folders or files. Read-only.
-    """
-    ignored_pairs = get_ignored_duplicate_pairs()
-    groups = []
-    for course_dir in get_all_course_dirs():
-        by_folder: Dict[Path, List[Path]] = {}
-        for f in course_dir.rglob('*'):
-            if f.is_file() and f.suffix.lower() in VIDEO_EXTENSIONS | AUDIO_EXTENSIONS:
-                by_folder.setdefault(f.parent, []).append(f)
-
-        for files in by_folder.values():
-            if len(files) < 2:
-                continue
-            entries = [{'path': f, 'tokens': _tokenize(f.stem)} for f in files]
-            n = len(entries)
-            for i in range(n):
-                for j in range(i + 1, n):
-                    path_i, path_j = str(entries[i]['path']), str(entries[j]['path'])
-                    if tuple(sorted((path_i, path_j))) in ignored_pairs:
-                        continue
-                    a, b = entries[i]['tokens'], entries[j]['tokens']
-                    union_tokens = a | b
-                    if not union_tokens:
-                        continue
-                    similarity = len(a & b) / len(union_tokens)
-                    if similarity < min_similarity:
-                        continue
-                    try:
-                        size_i = entries[i]['path'].stat().st_size
-                        size_j = entries[j]['path'].stat().st_size
-                    except OSError:
-                        continue
-                    groups.append({
-                        'course_name': course_dir.name,
-                        'course_path': str(course_dir),
-                        'files': sorted([
-                            {'path': path_i, 'name': entries[i]['path'].name, 'bytes': size_i, 'human': _format_bytes(size_i)},
-                            {'path': path_j, 'name': entries[j]['path'].name, 'bytes': size_j, 'human': _format_bytes(size_j)},
-                        ], key=lambda f: f['name'].lower()),
-                        'similarity': round(similarity, 2),
-                    })
-
-    groups.sort(key=lambda g: g['similarity'], reverse=True)
-    return groups
-
-
 def _find_stale_references() -> Dict[str, List[Dict[str, str]]]:
     """
     Entries in hidden_paths.json / next_up.json / recent_views.json that
@@ -3760,47 +3702,6 @@ def undo_last_action_api():
     })
 
 
-@app.route('/api/library/duplicate-lessons')
-def duplicate_lesson_files_api():
-    """Scan for lesson files within the same course/folder that look like
-    the same lesson downloaded twice (see _find_duplicate_lesson_files).
-    Manual trigger, same reasoning as Duplicate Courses."""
-    groups = _find_duplicate_lesson_files()
-    return jsonify({'groups': groups})
-
-
-@app.route('/api/library/delete-file', methods=['POST'])
-def delete_library_file_api():
-    """
-    Permanently delete a single media file from disk - the file-level
-    counterpart to /api/library/delete, used by Duplicate Lesson Files.
-    Scoped to files (not directories) inside the library root with a
-    recognized media extension, so it can't be pointed at something else.
-    """
-    data = request.json or {}
-    path = (data.get('path') or '').strip()
-    if not path:
-        return jsonify({'success': False, 'error': 'Missing path'}), 400
-
-    library_root = os.path.abspath(get_library_root())
-    target_abs = os.path.abspath(path)
-
-    if not target_abs.startswith(library_root + os.sep):
-        return jsonify({'success': False, 'error': 'Path is outside the library'}), 403
-    if not os.path.isfile(target_abs):
-        return jsonify({'success': False, 'error': 'No longer exists - already deleted?'}), 404
-    if Path(target_abs).suffix.lower() not in VIDEO_EXTENSIONS | AUDIO_EXTENSIONS:
-        return jsonify({'success': False, 'error': 'Not a media file'}), 400
-
-    try:
-        os.remove(target_abs)
-    except OSError as e:
-        return jsonify({'success': False, 'error': str(e)}), 500
-
-    invalidate_cache()
-    return jsonify({'success': True})
-
-
 @app.route('/api/library/duplicates')
 def duplicate_courses_api():
     """Scan for courses that look like copies of each other by name (see

templates/unsorted.html

@@ -687,16 +687,6 @@
             <div id="dup-results"></div>
         </div>
 
-        <h2 class="section-title">{{ icons.icon('clipboard', 18) }} Duplicate Lesson Files</h2>
-        <p class="section-desc">Scan for lesson files sitting in the same folder that look like the same lesson downloaded twice - a re-download that landed alongside the original instead of replacing it. Compares files within each folder only, with a higher match threshold than Duplicate Courses, so two genuinely different lessons on a similar topic aren't flagged.</p>
-        <div class="card">
-            <div class="toolbar">
-                <button class="btn btn-secondary" id="dup-lessons-scan-btn" onclick="scanDuplicateLessons()">{{ icons.icon('refresh', 14) }} Scan for Duplicate Lessons</button>
-                <span id="dup-lessons-status"></span>
-            </div>
-            <div id="dup-lessons-results"></div>
-        </div>
-
         <h2 class="section-title">{{ icons.icon('trash', 18) }} Clean Up Stale References</h2>
         <p class="section-desc">Renaming, moving, or deleting a course directly on the NAS (instead of through this app) can leave behind references in Hidden, Next Up, or Recently Viewed that point at nothing. Review and remove them here - course files on disk are never touched.</p>
         <div class="card">
@@ -1943,115 +1933,6 @@
             .catch(() => { btnEl.disabled = false; });
         }
 
-        // ---- Duplicate Lesson Files ----
-        function scanDuplicateLessons() {
-            const btn = document.getElementById('dup-lessons-scan-btn');
-            const status = document.getElementById('dup-lessons-status');
-            const results = document.getElementById('dup-lessons-results');
-            btn.disabled = true;
-            status.style.color = 'var(--text-muted)';
-            status.textContent = 'Scanning…';
-            results.innerHTML = '';
-            fetch('/api/library/duplicate-lessons')
-                .then(r => r.json())
-                .then(data => {
-                    btn.disabled = false;
-                    const groups = data.groups || [];
-                    if (!groups.length) {
-                        status.style.color = 'var(--success)';
-                        status.textContent = 'No likely duplicate lesson files found.';
-                        return;
-                    }
-                    status.style.color = 'var(--text-muted)';
-                    status.textContent = `${groups.length} possible duplicate group${groups.length === 1 ? '' : 's'}.`;
-                    results.innerHTML = groups.map(group => `
-                        <div class="dup-group">
-                            <div class="dup-group-header">
-                                <span>${escapeHtml(group.course_name)} — ${Math.round(group.similarity * 100)}% similar</span>
-                                <button class="btn btn-secondary btn-sm" onclick="ignoreDuplicateLessonGroup(this)">Not a duplicate - ignore</button>
-                            </div>
-                            ${group.files.map(f => `
-                                <div class="dup-course-row" data-path="${escapeAttr(f.path)}">
-                                    <span class="dup-course-name">${escapeHtml(f.name)}</span>
-                                    <span class="dup-course-path">${escapeHtml(f.human)}</span>
-                                    <button class="btn btn-danger btn-sm" onclick="deleteDuplicateLessonFile(this, '${f.path.replace(/'/g, "\\'")}', '${f.name.replace(/'/g, "\\'")}')">Delete</button>
-                                </div>
-                            `).join('')}
-                        </div>
-                    `).join('');
-                })
-                .catch(() => {
-                    btn.disabled = false;
-                    status.style.color = 'var(--error)';
-                    status.textContent = 'Could not reach the server.';
-                });
-        }
-
-        function deleteDuplicateLessonFile(btnEl, path, name) {
-            if (!confirm(`Permanently delete "${name}"?\n\n${path}\n\nThis cannot be undone.`)) return;
-            btnEl.disabled = true;
-            fetch('/api/library/delete-file', {
-                method: 'POST',
-                headers: { 'Content-Type': 'application/json' },
-                body: JSON.stringify({ path })
-            })
-            .then(r => r.json().then(data => ({ ok: r.ok, data })))
-            .then(({ ok, data }) => {
-                if (!ok || !data.success) {
-                    btnEl.disabled = false;
-                    alert(data.error || 'Delete failed');
-                    return;
-                }
-                const row = btnEl.closest('.dup-course-row');
-                const group = row ? row.closest('.dup-group') : null;
-                if (row) row.remove();
-                if (group && group.querySelectorAll('.dup-course-row').length < 2) {
-                    group.remove();
-                }
-                refreshDupLessonsStatusText();
-            })
-            .catch(() => {
-                btnEl.disabled = false;
-                alert('Could not reach the server.');
-            });
-        }
-
-        function refreshDupLessonsStatusText() {
-            const status = document.getElementById('dup-lessons-status');
-            const remaining = document.querySelectorAll('#dup-lessons-results .dup-group').length;
-            if (remaining === 0) {
-                status.style.color = 'var(--success)';
-                status.textContent = 'No possible duplicates remaining.';
-            } else {
-                status.style.color = 'var(--text-muted)';
-                status.textContent = `${remaining} possible duplicate group${remaining === 1 ? '' : 's'} remaining.`;
-            }
-        }
-
-        function ignoreDuplicateLessonGroup(btnEl) {
-            const group = btnEl.closest('.dup-group');
-            if (!group) return;
-            const paths = Array.from(group.querySelectorAll('.dup-course-row')).map(el => el.dataset.path);
-            if (paths.length < 2) return;
-            if (!confirm('Mark these as NOT duplicates of each other? They won\'t be flagged together again (reversible from "Ignored matches" under Duplicate Courses).')) return;
-            btnEl.disabled = true;
-            fetch('/api/library/duplicates/ignore', {
-                method: 'POST',
-                headers: { 'Content-Type': 'application/json' },
-                body: JSON.stringify({ paths })
-            })
-            .then(r => r.json())
-            .then(data => {
-                if (data.success) {
-                    group.remove();
-                    refreshDupLessonsStatusText();
-                } else {
-                    btnEl.disabled = false;
-                }
-            })
-            .catch(() => { btnEl.disabled = false; });
-        }
-
         // ---- Stale References ----
         let lastStaleItems = [];