LLMs <3 MIR

A tutorial on Large Language Models for Music Information Retrieval.

By Keunwoo Choi, 2024

About the Book

This is a web book I wrote because it felt fun when I thought about it – a tutorial on Large Language Models for Music Information Retrieval.

I’ve been working on acoustics, audio, music, machine learning, and AI since 2009 through various universities and companies. Early 2023, I decided to make a change towards language models and joined Genentech (Prescient Design team) to train and suffer from large language models. A year has passed, and I was fortunate to have learned some bit about LLMs by implementing, testing, evaluating, deploying, debugging, thinking, observing, wondering, and nightmare-ing.

LLMs seem powerful, but that may not be a strong reason to commit to them. But I believe LLMs are here to stay for good reasons. This book is my invitation to the field and the ideas, hope you like it.

The Scope

This book is written in the perspective of music AI.

1. Chapter I, “Large Language Models”, would be general and succinct. I’ll outsource a lot by simply sharing links so that you decide the depth and breadth of your study.

2. Chapter II, “LLM as a Tool with Common Sense” is where I introduce some existing works and my suggestions on how to use LLMs for MIR research.

3. Chapter III, “Multimodal LLMs”, provides a summary about how we can incorporate multimodal data into LLMs.

4. Chapter IV, “Weakness of LLMs for MIR”, presents some limitations the current LLMs have in the context of MIR research.

5. Chapter V, “Finale”, is just a single page of my parting words.

Ok, then..

Let’s go 🥁

---

Table of contents

I. Large Language Models

• 1. Basics: Before the LLMs
• 2. Architectures
• 3. Tokenizers
• 4. Pre-Training, and the Induced Properties
• 5. Finetuning
• 6. Evaluation

II. LLM as a Tool with Common Sense

• 1. LLMs are Zero-shot Learners
• 2. Dataset Cleaning and Creation
• 3. Music Information Extraction
• 4. Music Recommendation
• 5. Music Composition Assistance

III. Multimodal LLMs

• 1. Multimodal LLMs
• 2. Multimodal Input Processing
• 3. Music Audio LLMs

IV. Weakness of LLMs for MIR

• 1. Weaknesses of LLMs for MIR
• 2. LLMs can’t rhyme
• 3. LLMs don’t know every artist

V. Finale

• 1. Finale

Resources

• References

How to cite

@book{llms-heart-mir,
    Author = {Choi, Keunwoo},
    Month = May.,
    Title = {LLMs heart MIR: A tutorial on Large Language Models for Music Information Retrieval},
    Year = 2024,
    Url = {https://llms-heart-mir.github.io/tutorial},
}