Note: this is recommended reading for hardcore iTunes users only (and geeks).

Before I explain iTunes metadata, let me just first say that not all metadata are created equal and so some are backed up, some are not. I will go further into this discourse after an explanation. (AFAIK, there is no other online backup provider that backs up metadata the way Bandwagon does.)

Metadata explained

An item is composed usually of a track and an artwork. The track and the artwork are further composed of data also known as metadata (aka “propertiesâ€? in iTunes parlance). Below is the list of metadata for a track and an artwork (taken from iTunes Suite in Script Editor ¹ ).

track

• album (Unicode text) : the album name of the track
• album artist (Unicode text) : the album artist of the track
• artist (Unicode text) : the artist/source of the track
• title / name (taken from iTunes) ¹
• bit rate (integer, r/o) : the bit rate of the track (in kbps)
• bookmark (small real) : the bookmark time of the track in seconds
• bookmarkable (boolean) : is the playback position for this track remembered?
• bpm (integer) : the tempo of this track in beats per minute
• category (Unicode text) : the category of the track
• comment (Unicode text) : freeform notes about the track
• compilation (boolean) : is this track from a compilation album?
• composer (Unicode text) : the composer of the track
• database ID (integer, r/o) : the common, unique ID for this track. If two tracks in different playlists have the same database ID, they are sharing the same data.
• date added (date, r/o) : the date the track was added to the playlist
• description (Unicode text) : the description of the track
• disc count (integer) : the total number of discs in the source album
• disc number (integer) : the index of the disc containing this track on the source album
• duration (small real, r/o) : the length of the track in seconds
• enabled (boolean) : is this track checked for playback?
• episode ID (Unicode text) : the episode ID of the track
• episode number (integer) : the episode number of the track
• EQ (Unicode text) : the name of the EQ preset of the track
• finish (small real) : the stop time of the track in seconds
• gapless (boolean) : is this track from a gapless album?
• genre (Unicode text) : the music/audio genre (category) of the track
• grouping (Unicode text) : the grouping (piece) of the track. Generally used to denote movements within a classical work.
• kind (Unicode text, r/o) : a text description of the track
• long description (Unicode text)
• lyrics (Unicode text) : the lyrics of the track
• modification date (date, r/o) : the modification date of the content of this track
• played count (integer) : number of times this track has been played
• played date (date) : the date and time this track was last played
• podcast (boolean, r/o) : is this track a podcast episode?
• rating (integer) : the user rating of this track (0 to 100)
• sample rate (integer, r/o) : the sample rate of the track (in Hz)
• season number (integer) : the season number of the track
• shufflable (boolean) : is this track included when shuffling?
• skipped count (integer) : number of times this track has been skipped
• skipped date (date) : the date and time this track was last skipped
• show (Unicode text) : the show name of the track
• sort album (Unicode text) : override string to use for the track when sorting by album
• sort artist (Unicode text) : override string to use for the track when sorting by artist
• sort album artist (Unicode text) : override string to use for the track when sorting by album artist
• sort name (Unicode text) : override string to use for the track when sorting by name
• sort composer (Unicode text) : override string to use for the track when sorting by composer
• sort show (Unicode text) : override string to use for the track when sorting by show name
• size (integer, r/o) : the size of the track (in bytes)
• start (small real) : the start time of the track in seconds
• time (Unicode text, r/o) : the length of the track in MM:SS format
• track count (integer) : the total number of tracks on the source album
• track number (integer) : the index of the track on the source album
• unplayed (boolean) : is this track unplayed?
• video kind (none/movie/music video/TV show) : kind of video track
• volume adjustment (integer) : relative volume adjustment of the track (-100% to 100%)
• year (integer) : the year the track was recorded/released

artwork

• data (picture) : data for this artwork, in the form of a picture
• downloaded (boolean, r/o) : was this artwork downloaded by iTunes?
• format (type class, r/o) : the data format for this piece of artwork
• kind (integer) : kind or purpose of this piece of artwork

The most common types of metadata that we see when we use iTunes are album, title, artist, time, etc. These types are the most important to an “average� user. But for other users, the other types of metadata are just as important. They are used for sorting, creating smart playlists, or generally just being OC about their music data.

One of the most important decisions we made before we coded Bandwagon was the requirement of saving metadata as we are music lovers ourselves and thought this data is important. However, once we delved into the technical details, it became pretty challenging.

Here’s how we did it

Since metadata can be changed, certain date properties need to be updated when those types of metadata are changed. So metadata falls into two event properties: events affecting modification date and events affecting played date

a) modification date

• album
• album year
• artist
• title / name
• bit rate
• bpm
• composer
• comments
• date added
• disc count
• disc number
• duration
• episode ID
• episode number
• genre
• modification date
• season number
• show name
• size
• track count
• track number

b) played date

• played count
• played date

Bandwagon determines what metadata to backup using rules. So here are the rules for the respective date events for metadata backup.

a) modification date

If the track’s modification date changes, the following metadata gets updated on our indexers:

• album
• album artist
• album year
• artist
• title / name
• bpm
• comments
• compilation
• composer
• date added
• disc count
• disc number
• duration
• episode ID
• episode number
• equalizer settings
• finish
• genre
• grouping
• kind
• modification date
• played count
• played date
• season number
• show
• size
• skipped count
• skipped date
• sort album
• sort artist
• sort album artist
• sort name
• sort composer
• sort show
• start
• track count
• track number
• volume adjustment
• year

b) played date

If the track’s played date changes, the following metadata gets updated on our indexers:

• played count
• rating
• played date

Caveat: data (artwork) and lyrics are backed up once only (if you add artwork or lyrics to an already backed up track, it will NOT be backed up)

The following list are metadata that Bandwagon does not backup.

The main reasons we don’t backup the following metadata list are two-fold:

1. unreadable

• bookmark
• category
• description
• long description
• shufflable

2. no need backing up (e.g. time is just the MM:SS format for duration. If duration exists, time exists.)

• bit rate
• bookmarkable
• database ID
• enabled
• gapless
• podcast
• sample rate
• time
• unplayed
• video kind
• downloaded (artwork)
• format (artwork)
• kind (artwork)

For most iTunes users, we think that, while the unreadable metadata is still data, it’s not as important as the others (some will disagree, we know). However, we will keep looking for ways to back up the unreadables. If you have any ideas, send them our way support@ridethebandwagon.com

Hi Terence!

First, I have a request/suggestion about the modification vs. played date: I believe that rating should belong in both places, as it’s possible that a listener could be scrolling through a playlist and notice an unrated track. “Oh, I haven’t rated that one? That’s a 4.” Click. The rating doesn’t get recorded because it doesn’t get played; the user knows the song well.

on get_track_category(the_dbID)
        tell application "iTunes" 
            repeat with t in every file track of the library playist
            -- presumably, you'd have a much smaller list to traverse
            -- not the entire library
                if the database ID of t equals the_dbID then
                    set theCategory to the category of t
                end if
            end repeat
        end tell
    return theCategory
end get_track_category

Then, inside your Cocoa app, you can call the script with OSAExecuteEvent. Info on doing so is at QA1111.

I’ve never used the category myself (since it seems so difficult to access; why’d Apple do that?!) but I’d like to, once I make it easier to do so via scripting. I’m already putting other types of metadata into the composer and grouping fields besides what’s supposed to go there. Category would be much more appropriate.

Hey Billifer,

1) RE: rating – yeah, we agree with you but there is no easy way to do this via the modification date as played date is actually NOT a subset of modification date. As there is no event that will tell us when rating has been randomly clicked, we don’t have too many choices. Re-scanning the XML is a possibility but that’s resource intensive for large libraries.

2) RE: rest of the metadata – we actually had a number of AppleScripts that grabbed the hard-to-get metadata but they just did not scale for large libraries so we had to abandon them.

Thanks for your thoughtful suggestion and sample AS :)

In addition to Terence’ reply, not only that it’s so slow gathering metadata through applescript, but also the fact that interfacing between Applescript and Cocoa is very buggy back when we were doing Xacktunes. We really have to decide whether to use Applescript or not, and deciding not to be very dependent on Applescript (we only use it for importing files) makes everything in Bandwagon faster by a tenfold. Not to mention that you have to have iTunes running just to get your desired data.

In the future, maybe on v1.5, we’re planning to use applescript on a more optimized way. By that time, the index is still as compatible as the indexer now. Or in case we gain access to the proprietary itl file, there will be no need tinkering with the XML library. That’s the best approach I can think of.

One thing about AppleScript is that it does not scale well. It’s a behemoth. There’s always the QuickTime 7 framework metadata access routines. :-D

I’m sure you’ve considered all this, but perhaps an iTunes plugin that acts as a client to a BW app “server” could be a viable option; there’s at least some published information on how to write plugins/bundles that can do all that work from inside iTunes.

The other thing I’m thinking of is this: As files are added to the backup queue, add a new Spotlight metadata item to them, like kMDItemBWCategory, and then copy the category (if it exists) into the Spotlight metadata. Since that MD is stored in the resource fork of the file, it’s going to be easier to access (tools like mdls, mdfind, etc.), and it’ll tag along with the data too, provided you’re not tossing away the resource fork when you do the backup. I imagine, though, that will be partly storage-dependent, as FTP servers in particular will not like OS X-style resource forks and at best will store each file as, say, ABC123 and ._ABC123.

Just some other ideas, from a different perspective, for you to consider. :) Keep up the good work!

Updated: any changes to any metadata will now be recognized in “near real-time” due to a change in our algo. This occurs anytime your itunes xml changes.