Allows us to have custom release names per view and avoids a mutable identifier flying around everywhere.

• Update internal APIs and CLIs
• Update documentation
• Replace ghost_empty_writeable_directories with something more akin to the file creation special ops.
• Delete concept in cache + elsewhere
• Expand the VirtualPath abstraction to encapsulate generating paths & mapping from views

We don't care about memory, and it's in the hot path of virtual filesystem caching. We should go as close to native as possible.

warn if the image in tags differs from the image on disk; otherwise always rm embed and make external

this ticket covers the items from init -> 0.3, as creating separate issues is kind of silly for the initial bootstrap

browser:

• tag reading
• initial cache population command
• read ops virtual fs
• preliminary README
• cover.{jpg,png} support
• json dump for jq querying
• artist/genre/label hiding (from vfs)
• collages
• writes
• watchdog
• new albums toggling
• added_at sorted view
• subartists / artist aliases
• persistent track IDs
• playlists
  • add to playlist via filesystem...
  • playlist cover art
• support arbitrarily nested release directories (replace scandir w/ walk)
• collage/playlist description_meta autoupdating
• docs dir (unbundle readme; too unorganized)
• trigger cache update after release call in virtual fs
• add/modify/delete cover art via virtual fs. accept writes to known valid cover art filenames and ghost file rename

metadata mgmt:

• tag writing
• album file editing as text file
• add validation to album file editing for release type artist roles
• click autocomplete
• rules engine
  • single-matcher rule execution (char normalization / artist,label,genre rename / tag splitting) / stored file
  • act on other tag (e.g. artist:beethoven role=composer)
  • multiple actions
  • dsl for shell + cli editing
  • clean up UI
  • optimize - sqlite fts5
  • cli
  • dry-run flag
  • doc
  • add (for multi-value tags, add a value, like a genre)
• single extractor (extract a song as a single; adjust album-level tags+releasetype)

cleanup:

• virtualfs tests
• replace migrations system with db nuke on mig hash change
• factor artist parsing/formatting semantics into its own package
• no discno in single tag files
• bug: preview-tui syscalls missing
• add more data to stat - mtimes!
• flatten dirs / move testdatas into $root/testdata/{cache,tagger}
• move to per-release .rose.toml that contains uuid+new; then use discno/trackno for playlists
• optimize FUSE (it's 100x slower rn than native, wtf??)
• cache refresh optimization time: possibly mtime tracking w/ --force override
• write logs to ~/.local/smthn/rose
• emit log lines for each release deleted b/c no longer on disk (RETURNING supported by sqlite?)
• error.py -> base_error.py
• bug: unnecessary cache miss for Fading Up collage?
• push sanitize into cache module and handle it all there; return (raw, sanitized) dataclasses
• optimize cold start cache by batching disk writes after the loop
• parallelize cache updates
• config parsing error handling with good messages
• bug: where artists/genre/labels deleted from tags are not evicted from cache
• bug: multiprocessing error propagation from subproces
• invalidate cache on config change
• optimize watcher test
• improve type of artist aliases; it turns out toml has dicts..
• refactor fuse_hide_x configs to a combo of fuse_x_{white,black}list
• sqlite locks w/ timeouts for concurrency primitive? i hate worrying about concurrent execution
• collage/playlist update concurrency safety
• toggle new concurrency safety
• move supported ext and rlstypes to tagger. move supported roles to artiststr.
• optimize playlist addition via virtualfs (i didnt realize there would be so many calls to write, we are reallocing way too many times)
• bug: renaming rls/track while track is in a playlist results in track disappearing from playlist until forced cache update
• feature: cover art filenames as config array
• improvement: config value for ignore_directories
• bug: release editor sorts tracks incorrectly: "10" sorts before "2"
• bug: virtual filenames are not unique during playlist editing: the remapping can change IDs if two tracks have the same virtual filename. add temporary collision ids
• bug: rename playlist art when renaming a playlist
• feature: support parsing release virtual dirname out of path type for CLI args
• bug: audit collage/playlist mechanics when releases are deleted/replaced during update (e.g. during dir rename)
• improvement: before editing any metadata or something, do a mtime check, and a cache update if the mtimes dont match. so that rose doesnt cause a lost write.
• perf: optimize the hell out whatever's making ls slow
• refactor: switch to https://github.com/libfuse/pyfuse3; fusepy is 5 years w/out update. UPDATE: we went with llfuse instead.
• rose cache unwatch and pid/de-duplicate detection
• improvement: log rotation
• bug: standard tools error in the vfs "add to playlist/collage" operation. update vfs to pretend to be normal in these situations within a time window (e.g. allow file writing when file writing actually shouldnt be allowed)
• bug: an invalid rename operation can permalock parent directory. might be a fusepy bug; reevaluate after switch to pyfuse3
• bug: collage vfs adder doesnt work if directories are copied from another collage or recently added, because virtual dirname has prefixes. soln: strip a few prefix regexes when attempting to look up...
• compat: mac support (should only require supporting alts. for XDG_*)
• do not hard delete missing music from collage/playlist; instead, add missing field
• renaming {NEW} via virtual fs crashes in the cache update... writes the file, cache fails to refresh dirname, vfs fails
• improvement: write rose_release_ids to audio tags too in order to make watchdog updates more robust and prevent double .rose.{uuid}.toml writes
• process: branching strategy: have release be the stable branch; master be the unstable branch? update release to the latest release version
• optimization: optimize cache again...
• robustness: if release ID in tags, don't process unless force passed, emit explicit warnings
• bug: release cache doesnt invalidate if a track is deleted from a release
• don't schedule duplicate playlist updates (returnval propagate)

docs:

• ingestion: tag splitting behavior
• blacklist/whitelist use case in virtual filesystem
• single extraction

it can use the rules engine internally

Depends on #7. #7 makes rest of system agnostic to virtual dirnames.

Then move artiststr into audiotags: we can use more stylistically appealing formatting in the filesystem, e.g. x, y & z multi-value delimiting.

Use Jinja templates (\s+ -> space); have a test command that prints the templates with demo values, "fs preview-templates"

• template support
• user-configurable templates
• preview command
• documentation

support flags:

tags:pattern:i
tags:pattern:i::replace:value

i can't figure this out for the life of me. all the multiprocessing code (cache.py, virtualfs.py, watcher.py) does not record coverage information. gpt-4 is not useful, and https://pytest-cov.readthedocs.io/en/latest/subprocess-support.html produces zero effects.

help appreciated!

Support specifying scripts to be invoked at certain points, e.g. cache update events or on release edit or something else. Rose can be extended by these scripts

makes it easier to script against them without resorting to miserable jq tricks. also more performant because those views wont grab tracks

Hitch a ride on #11 templating language once it's there

A rare need, which is why it's been punted to its own ticket, but nice to have for interface completeness

perhaps a part of ext? or mainline?

i want the [P] and [C] prefixes but that's because of my mobile player...

#!/usr/bin/env python

"""
This script takes every Rose .toml playlist and creates a corresponding .m3u8 playlist
file for my phone. This script does the same for collages too.
"""

import logging
import sys
import subprocess
import contextlib
import sqlite3
from pathlib import Path
from typing import Iterator

logger = logging.getLogger()
logger.setLevel(logging.INFO)
stream_formatter = logging.Formatter(
    "[%(asctime)s] %(levelname)s: %(message)s",
    datefmt="%H:%M:%S",
)
stream_handler = logging.StreamHandler(sys.stdout)
stream_handler.setFormatter(stream_formatter)
logger.addHandler(stream_handler)


MUSIC_SOURCE_DIR = Path.home() / '.music-source'
PLAYLISTS_DIR = MUSIC_SOURCE_DIR / '!playlists'
COLLAGES_DIR = MUSIC_SOURCE_DIR / '!collages'
MOBILE_SOURCE_PATH = "/primary/music-synced"
DATABASE_PATH = Path.home() / '.cache' / 'rose' / 'cache.sqlite3'

# Copy pasted from Rose.
@contextlib.contextmanager
def connect() -> Iterator[sqlite3.Connection]:
    conn = sqlite3.connect(
        str(DATABASE_PATH),
        detect_types=sqlite3.PARSE_DECLTYPES,
        isolation_level=None,
        timeout=15.0,
    )
    try:
        conn.row_factory = sqlite3.Row
        conn.execute("PRAGMA foreign_keys=ON")
        conn.execute("PRAGMA journal_mode=WAL")
        yield conn
    finally:
        if conn:
            conn.close()


# Update the cache before we zoom!
subprocess.run(["rose", "cache", "update"], check=True)
logger.info("Finished updating Rose's cache!")

# Convert playlists.
for p in PLAYLISTS_DIR.iterdir():
    if p.suffix == ".m3u8":
        p.unlink()
for p in PLAYLISTS_DIR.iterdir():
    if not p.suffix == ".toml":
        continue

    logger.info(f"Converting playlist {p.name}")

    with connect() as conn:
        cursor = conn.execute(
            """
            SELECT t.source_path
            FROM tracks t
            JOIN playlists_tracks pt ON pt.track_id = t.id
            WHERE pt.playlist_name = ?
            ORDER BY pt.position ASC
            """,
            (p.stem,),
        )
        source_paths = [r["source_path"] for r in cursor]
        logger.debug(f"Found source paths: {source_paths}")

    mobile_paths = [
        MOBILE_SOURCE_PATH + p.removeprefix(str(MUSIC_SOURCE_DIR)) for p in source_paths
    ]
    logger.debug(f"Converted to mobile paths: {mobile_paths}")

    m3u8_file = p.with_name('[P] ' + p.name).with_suffix('.m3u8')
    with m3u8_file.open("w") as fp:
        fp.write("#EXTM3U\n")
        for p in mobile_paths:
            fp.write(f"{p}\n")

    logger.info(f"Wrote {m3u8_file.name}!")

# Convert playlists.
for p in COLLAGES_DIR.iterdir():
    if p.suffix == ".m3u8":
        p.unlink()
for p in COLLAGES_DIR.iterdir():
    if not p.suffix == ".toml":
        continue

    logger.info(f"Converting collage {p.name}")

    with connect() as conn:
        cursor = conn.execute(
            """
            SELECT t.source_path
            FROM tracks t
            JOIN releases r ON r.id = t.release_id
            JOIN collages_releases cr ON cr.release_id = r.id
            WHERE cr.collage_name = ?
            ORDER BY cr.position ASC, t.formatted_release_position ASC
            """,
            (p.stem,),
        )
        source_paths = [r["source_path"] for r in cursor]
        logger.debug(f"Found source paths: {source_paths}")

    mobile_paths = [
        MOBILE_SOURCE_PATH + p.removeprefix(str(MUSIC_SOURCE_DIR)) for p in source_paths
    ]
    logger.debug(f"Converted to mobile paths: {mobile_paths}")

    m3u8_file = p.with_name('[C] ' + p.name).with_suffix('.m3u8')
    with m3u8_file.open("w") as fp:
        fp.write("#EXTM3U\n")
        for p in mobile_paths:
            fp.write(f"{p}\n")

    logger.info(f"Wrote {m3u8_file.name}!")

Sanitization.. haha

e.g. rose release run-rule {release_id} 'action1' 'action2'
and analogous for tracks

no initial matcher; only using the action matcher
this is for shell scripting

again... the jinja templates tanked the performance. caching is not much of a perf improvement due to key construction expense. i can fix it this time, but this is going to keep popping up

future thoughts:

1. benchmark at 10k releases. i've only ran with 1k, but i don't think the virtual filesystem will scale to 10k well. luckily the rules engine should (thanks sqlite).
2. add performance tests of some kind? this is a real regression that materially affects the user experience.
3. rewrite virtual filesystem in zig or rust. maybe rewrite it all later.

We can use the metadata rules engine instead to implement "splitting" of other tags as-needed. I'm comfortable with ;, but it's not great as some vaporwave artists might use ;.

Scared anal people can use \\ or some insane sequence, or we can potentially even support "multi-frame" to be truly correct!

potential race condition bug

This would allow mpv to continue accessing and playing tracks even if we renamed a directory. Modifying metadata during playback is messing with playback when the virtual paths change, and this works around that.

This is best implemented as an extension of VirtualPath once #7 is implemented.

Add a RoseExpectedError subclass that user-facing expected errors should subclass. These errors do not print their traceback (which we can implement in __main__.py. Other errors continue to print their traceback.

Validation errors should be of this kind.

e.g. tracktitle: empty pattern

trackartist[role] albumartist[role]

and then ban use of replaceall on the no-role generalized artist matcher

Mutations still work. But the virtual filesystem would not use the {NEW} prefix

Not important, since we update the cache before performing release edit, but it's better to not use the cache at all in the write path.

Better architecturally 👍🏼 Also means we do not have to invalidate the cache on alias change

• Read cache database schema
• Cached* structs
• Metadata* structs
• AudioTags struct
• Rule engine syntax

Standardize on the rule engine field names.

Not too much we can do about a cold cache, as that has a baseline that's probably around 600-800ms for 1k releases in Python.

However, a hot cache should be able to be no-opped extremely cheaply.

Notepad for future improvements:

• Don't pull all releases and all tracks at the start. Only pull the minimal amount of data needed to check whether we need to even upsert, which are mtimes and cover image path. Batch pull the full cache data only for the releases that need an update
• Stop doing string transformations in tight inner loops
• Skip logics like the dirname calcs if dirty = False

• add links
• metadata import given link from:
  • discogs
  • musicbrainz
  • tidal
  • deezer
  • itunes/apple
  • junodownload
  • beatport
  • rym
• cover art download given link for:
  • discogs
  • musicbrainz
  • tidal
  • deezer
  • itunes/apple
  • fanart.tv
  • junodownload
  • beatport
  • rym
• search for link in:
  • discogs
  • musicbrainz
  • tidal
  • deezer
  • itunes/apple
  • fanart.tv
  • junodownload
  • beatport

on top of only name

• Move current commands to print-all and add a print command that takes in UUID/path and prints one release
• Add rose tracks print
• Support matcher param to filter releases/tracks print-all. only allow release-level tags for release one
• Document!

And potentially update the list_releases calls in the vfs to use the rules engine matching... check perf penalty. If that works performantly, it's 1/2 the implementation of having "smart collages" and 1/3 of "smart playlists"

Right now, when the configuration changes, the cache is simply nuked whenever any Rose command is run, and the running virtual fs blanks. The config change should not be as disruptive to tool usage; this behavior is unintuitive and chaotic, going psychotic.

Potential solution: Run a cache update after database upgrade. And only "swap" the database once the new database is bootstrapped.

Keep current behavior if there is no database file that is getting upgraded.

Alao add a command that refreshes the config in any running Rose programs.

Create custom views of tracks and releases based on filters.

Stored smart playlists/collages are trivial. To make the experience ergonomic, we should have a config refresh option that reads pids and sends signal to refresh config.

Default filenames sorted Artist, Album, Track. Only makes sense after #7.

Because rm -r calls unlink on child directories, which triggers the file removal operations for all child files.

We should probably disable the ability to remove cover art by deleting the file.

It's been slapped together rather quickly with each feature: the content is mostly there or todo, but the writing style hurts a bit.