rmsitz/offlineu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove unreliable duplicate-lesson-file matching

Browse Source 
Filename-similarity matching within a folder can't distinguish a real
re-download duplicate from a course that splits one topic across
several numbered files ("...Part 1"/"...Part 2", "-1"/"-2"/"-3") -
both look like near-100% matches by name alone, and the latter is a
common, completely normal pattern. This produced hundreds of false
positives in practice. Removes Duplicate Lesson Files entirely
(detection function, its two API routes, and the File Management UI
section); whole-course-folder matching (Duplicate Courses) is
unaffected and stays the only duplicate scan in the app.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
rmsitz
2026-08-25 08:25:37 -04:00
co-authored by Claude Sonnet 5
parent 8220bb542a
commit 72c7f67c3d
5 changed files with 13 additions and 233 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+1
View File
@@ -1,3 +1,4 @@
2026-08-25 12:25 UTC — Remove unreliable duplicate-lesson-file matching
2026-08-25 00:33 UTC — Widen thumbnail sampling window past platform bumpers
2026-08-25 00:26 UTC — Fix race condition in thumbnail candidate generation
2026-08-25 00:20 UTC — Add autoplay, progress rings, duplicate-lesson detection, title-card thumbnails
README.md
+11 -14
View File
@@ -185,20 +185,17 @@ silently stay blank instead of erroring.
difference doesn't hide a real duplicate. Groups by similarity (union of
any two courses over the threshold, transitively) rather than showing
raw pairs; each course in a group gets a one-click Hide, or a **Delete**
that permanently removes it from disk (confirmed with the full path
before it runs). A group can also be marked **"Not a duplicate"** if the
match is wrong, which excludes that specific pair from future scans
without touching anything else that happens to match one of those
courses; ignored pairs are listed (and reversible) under "Ignored
matches," and persist through backup/restore alongside hidden paths and
Next Up.
- *Duplicate Lesson Files*: the same idea one level down - media files
sitting in the same folder that look like the same lesson downloaded
twice (e.g. a re-download that landed alongside the original instead of
replacing it). Compares only within a folder, at a higher match
threshold than Duplicate Courses, so two different lessons on a similar
topic don't get flagged; shares its ignore list with Duplicate Courses.
Delete here removes a single file, not a whole course.
that permanently removes it from disk (the one destructive action in the
whole app - confirmed with the full path before it runs). A group can
also be marked **"Not a duplicate"** if the match is wrong, which
excludes that specific pair from future scans without touching anything
else that happens to match one of those courses; ignored pairs are
listed (and reversible) under "Ignored matches," and persist through
backup/restore alongside hidden paths and Next Up. (Matching is
deliberately scoped to whole course folders, not individual lesson
files - courses that split one topic across several numbered files,
e.g. "...Part 1"/"...Part 2", look just as similar by name as a genuine
re-download, so file-level matching wasn't reliable enough to keep.)
- *Clean Up Stale References*: finds entries in the hidden-paths list,
Next Up queue, Favorites, or Recently Viewed history that point at a path
no longer on disk - normally from renaming/moving/deleting a course
VERSION
+1 -1
View File
@@ -1 +1 @@
2026-08-25 00:33 UTC — widen thumbnail sampling window past platform bumpers
2026-08-25 12:25 UTC — remove unreliable duplicate-lesson-file matching
offlineu_core.py
-99
View File
@@ -1864,64 +1864,6 @@ def _find_duplicate_courses(min_similarity: float = 0.6) -> List[Dict[str, Any]]
return groups
def _find_duplicate_lesson_files(min_similarity: float = 0.75) -> List[Dict[str, Any]]:
"""
Within each course, media files sitting in the same folder whose
names look like the same lesson downloaded twice - e.g. "03 Setting
Up Your Environment.mp4" and "03 Setting Up Your Environment (1).mp4"
left behind by a re-download that landed in the same section. Scoped
to siblings in the same folder, unlike Duplicate Courses' whole-library
comparison: two genuinely different lessons can share a lot of
vocabulary ("Part 1"/"Part 2" of a topic) without being duplicates, so
same-folder pairing plus a higher similarity threshold keeps this
conservative. Shares Duplicate Courses' ignore list (get_ignored_
duplicate_pairs) - a pair is just a pair of paths there, whether
they're course folders or files. Read-only.
"""
ignored_pairs = get_ignored_duplicate_pairs()
groups = []
for course_dir in get_all_course_dirs():
by_folder: Dict[Path, List[Path]] = {}
for f in course_dir.rglob('*'):
if f.is_file() and f.suffix.lower() in VIDEO_EXTENSIONS | AUDIO_EXTENSIONS:
by_folder.setdefault(f.parent, []).append(f)
for files in by_folder.values():
if len(files) < 2:
continue
entries = [{'path': f, 'tokens': _tokenize(f.stem)} for f in files]
n = len(entries)
for i in range(n):
for j in range(i + 1, n):
path_i, path_j = str(entries[i]['path']), str(entries[j]['path'])
if tuple(sorted((path_i, path_j))) in ignored_pairs:
continue
a, b = entries[i]['tokens'], entries[j]['tokens']
union_tokens = a | b
if not union_tokens:
continue
similarity = len(a & b) / len(union_tokens)
if similarity < min_similarity:
continue
try:
size_i = entries[i]['path'].stat().st_size
size_j = entries[j]['path'].stat().st_size
except OSError:
continue
groups.append({
'course_name': course_dir.name,
'course_path': str(course_dir),
'files': sorted([
{'path': path_i, 'name': entries[i]['path'].name, 'bytes': size_i, 'human': _format_bytes(size_i)},
{'path': path_j, 'name': entries[j]['path'].name, 'bytes': size_j, 'human': _format_bytes(size_j)},
], key=lambda f: f['name'].lower()),
'similarity': round(similarity, 2),
})
groups.sort(key=lambda g: g['similarity'], reverse=True)
return groups
def _find_stale_references() -> Dict[str, List[Dict[str, str]]]:
"""
Entries in hidden_paths.json / next_up.json / recent_views.json that
@@ -3760,47 +3702,6 @@ def undo_last_action_api():
})
@app.route('/api/library/duplicate-lessons')
def duplicate_lesson_files_api():
"""Scan for lesson files within the same course/folder that look like
the same lesson downloaded twice (see _find_duplicate_lesson_files).
Manual trigger, same reasoning as Duplicate Courses."""
groups = _find_duplicate_lesson_files()
return jsonify({'groups': groups})
@app.route('/api/library/delete-file', methods=['POST'])
def delete_library_file_api():
"""
Permanently delete a single media file from disk - the file-level
counterpart to /api/library/delete, used by Duplicate Lesson Files.
Scoped to files (not directories) inside the library root with a
recognized media extension, so it can't be pointed at something else.
"""
data = request.json or {}
path = (data.get('path') or '').strip()
if not path:
return jsonify({'success': False, 'error': 'Missing path'}), 400
library_root = os.path.abspath(get_library_root())
target_abs = os.path.abspath(path)
if not target_abs.startswith(library_root + os.sep):
return jsonify({'success': False, 'error': 'Path is outside the library'}), 403
if not os.path.isfile(target_abs):
return jsonify({'success': False, 'error': 'No longer exists - already deleted?'}), 404
if Path(target_abs).suffix.lower() not in VIDEO_EXTENSIONS | AUDIO_EXTENSIONS:
return jsonify({'success': False, 'error': 'Not a media file'}), 400
try:
os.remove(target_abs)
except OSError as e:
return jsonify({'success': False, 'error': str(e)}), 500
invalidate_cache()
return jsonify({'success': True})
@app.route('/api/library/duplicates')
def duplicate_courses_api():
"""Scan for courses that look like copies of each other by name (see
templates/unsorted.html
-119
View File
@@ -687,16 +687,6 @@
<div id="dup-results"></div>
</div>
<h2 class="section-title">{{ icons.icon('clipboard', 18) }} Duplicate Lesson Files</h2>
<p class="section-desc">Scan for lesson files sitting in the same folder that look like the same lesson downloaded twice - a re-download that landed alongside the original instead of replacing it. Compares files within each folder only, with a higher match threshold than Duplicate Courses, so two genuinely different lessons on a similar topic aren't flagged.</p>
<div class="card">
<div class="toolbar">
<button class="btn btn-secondary" id="dup-lessons-scan-btn" onclick="scanDuplicateLessons()">{{ icons.icon('refresh', 14) }} Scan for Duplicate Lessons</button>
<span id="dup-lessons-status"></span>
</div>
<div id="dup-lessons-results"></div>
</div>
<h2 class="section-title">{{ icons.icon('trash', 18) }} Clean Up Stale References</h2>
<p class="section-desc">Renaming, moving, or deleting a course directly on the NAS (instead of through this app) can leave behind references in Hidden, Next Up, or Recently Viewed that point at nothing. Review and remove them here - course files on disk are never touched.</p>
<div class="card">
@@ -1943,115 +1933,6 @@
.catch(() => { btnEl.disabled = false; });
}
// ---- Duplicate Lesson Files ----
function scanDuplicateLessons() {
const btn = document.getElementById('dup-lessons-scan-btn');
const status = document.getElementById('dup-lessons-status');
const results = document.getElementById('dup-lessons-results');
btn.disabled = true;
status.style.color = 'var(--text-muted)';
status.textContent = 'Scanning…';
results.innerHTML = '';
fetch('/api/library/duplicate-lessons')
.then(r => r.json())
.then(data => {
btn.disabled = false;
const groups = data.groups || [];
if (!groups.length) {
status.style.color = 'var(--success)';
status.textContent = 'No likely duplicate lesson files found.';
return;
}
status.style.color = 'var(--text-muted)';
status.textContent = `${groups.length} possible duplicate group${groups.length === 1 ? '' : 's'}.`;
results.innerHTML = groups.map(group => `
<div class="dup-group">
<div class="dup-group-header">
<span>${escapeHtml(group.course_name)} — ${Math.round(group.similarity * 100)}% similar</span>
<button class="btn btn-secondary btn-sm" onclick="ignoreDuplicateLessonGroup(this)">Not a duplicate - ignore</button>
</div>
${group.files.map(f => `
<div class="dup-course-row" data-path="${escapeAttr(f.path)}">
<span class="dup-course-name">${escapeHtml(f.name)}</span>
<span class="dup-course-path">${escapeHtml(f.human)}</span>
<button class="btn btn-danger btn-sm" onclick="deleteDuplicateLessonFile(this, '${f.path.replace(/'/g, "\\'")}', '${f.name.replace(/'/g, "\\'")}')">Delete</button>
</div>
`).join('')}
</div>
`).join('');
})
.catch(() => {
btn.disabled = false;
status.style.color = 'var(--error)';
status.textContent = 'Could not reach the server.';
});
}
function deleteDuplicateLessonFile(btnEl, path, name) {
if (!confirm(`Permanently delete "${name}"?\n\n${path}\n\nThis cannot be undone.`)) return;
btnEl.disabled = true;
fetch('/api/library/delete-file', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ path })
})
.then(r => r.json().then(data => ({ ok: r.ok, data })))
.then(({ ok, data }) => {
if (!ok || !data.success) {
btnEl.disabled = false;
alert(data.error || 'Delete failed');
return;
}
const row = btnEl.closest('.dup-course-row');
const group = row ? row.closest('.dup-group') : null;
if (row) row.remove();
if (group && group.querySelectorAll('.dup-course-row').length < 2) {
group.remove();
}
refreshDupLessonsStatusText();
})
.catch(() => {
btnEl.disabled = false;
alert('Could not reach the server.');
});
}
function refreshDupLessonsStatusText() {
const status = document.getElementById('dup-lessons-status');
const remaining = document.querySelectorAll('#dup-lessons-results .dup-group').length;
if (remaining === 0) {
status.style.color = 'var(--success)';
status.textContent = 'No possible duplicates remaining.';
} else {
status.style.color = 'var(--text-muted)';
status.textContent = `${remaining} possible duplicate group${remaining === 1 ? '' : 's'} remaining.`;
}
}
function ignoreDuplicateLessonGroup(btnEl) {
const group = btnEl.closest('.dup-group');
if (!group) return;
const paths = Array.from(group.querySelectorAll('.dup-course-row')).map(el => el.dataset.path);
if (paths.length < 2) return;
if (!confirm('Mark these as NOT duplicates of each other? They won\'t be flagged together again (reversible from "Ignored matches" under Duplicate Courses).')) return;
btnEl.disabled = true;
fetch('/api/library/duplicates/ignore', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ paths })
})
.then(r => r.json())
.then(data => {
if (data.success) {
group.remove();
refreshDupLessonsStatusText();
} else {
btnEl.disabled = false;
}
})
.catch(() => { btnEl.disabled = false; });
}
// ---- Stale References ----
let lastStaleItems = [];