NLP Tools

Preview

A preview of experiments and studies in preparing and measuring literary and lyric text for remixing.

PreviewOngoing experiments. The full interactive versions will launch in the LiteraryDJ web app.

Experiment01

Remix Corpus Pipeline

An experiment in preparing writers’ texts for remixing so every word can be traced back to its writer, work, and edition. It separates a writer’s words from publishers’ text and keeps their punctuation, spelling, and dialect.

Our pipeline

Provenance

  1. As written

    When justfeelings totell me

    Original

  2. Common tools

    whenjustfeelingstotellme

    ✕ Loseswhose words they were

  3. Our pipeline

    1. When just…how it would be / When just this time, some perfect year…
    2. feelings to…And to thine heart just feelings to convey…
    3. tell me…And the earth, they tell me, / On its axis turned…

    ✓ Keepseach fragment’s writer, poem, and edition

  • Dickinson
  • Wheatley
  • Both writers

A line composed on Markov Monostich from Emily Dickinson and Phillis Wheatley

Elisions

  1. As written

    Which wip’d the trick’ling tear from Mis’ry’s eye.

    Original

  2. Common tools

    whichwipdthetricklingtearfrommisryseye

    ✕ Losesthree words, broken into seven pieces

  3. Our pipeline

    Whichwip’dthetrick’lingtearfromMis’ry’seye

    ✓ Keepsevery word whole, as Wheatley spelled it

Phillis Wheatley, “To His Honour the Lieutenant-Governor, on the Death of His Lady” (1773)

Sentences

  1. As written

    Mr. Covey bought her from Mr. Thomas Lowe, about six miles from St. Michael’s.

    Original

  2. Common tools

    1. Mr.
    2. Covey bought her from Mr.
    3. Thomas Lowe, about six miles from St.
    4. Michael’s.

    ✕ Losesthe sentence: each title reads as a full stop

  3. Our pipeline

    1. Mr. Covey bought her from Mr. Thomas Lowe, about six miles from St. Michael’s.

    ✓ Keepsthe sentence, whole

Frederick Douglass, Narrative of the Life of Frederick Douglass (1845)

Line breaks

  1. As written

    ’Tis little I could care for pearlsWho own the ample sea;Or brooches, when the EmperorWith rubies pelteth me;

    Original

  2. Common tools

    1. ’Tis little I could care for pearls
    2. Who own the ample sea;
    3. Or brooches, when the Emperor
    4. With rubies pelteth me;

    ✕ Losesthe thought that runs from line to line

  3. Our pipeline

    1. ’Tis little I could care for pearls / Who own the ample sea; / Or brooches, when the Emperor / With rubies pelteth me; …

    ✓ Keepsthe sentence, across its line breaks

Emily Dickinson, “Real Riches,” Poems, Third Series (1896)

Experiment02

Obscurity Scores

An experiment behind the Obscurity dial in Markov Monostich: steering a remix toward words that were ordinary in nineteenth-century print but sound strange today, like o’er and thither.

Uses per million words of print

  • o’er1800–1930: 191990–2019: 2.1
  • thither1800–1930: 131990–2019: 2.0
  • withal1800–1930: 4.81990–2019: 0.7

Google Books Ngrams · CC BY 3.0

Experiment03

Markov Suitability

An experiment in judging whether a set of lyrics or poems is ready to remix, by counting how many different words can follow each word.

Share of words, by what can follow them

  • A few poems

    19% have a choice of next words81% have only one next word

  • The whole collection

    32% have a choice of next words68% have only one next word

Experiment04

Lyric Profiles

An experiment in visualizing a corpus of music lyrics. It measures lyrics in two layers, semantic and lexical, and groups the measures into the four qualities that shape each artist’s knot.

Concept & initial development

Created as coursework for MITH388 at the University of Maryland. Under continued development by the LiteraryDJ team.

Liner NotesPrototyping Lyrical Torus KnotsRead Arshad’s process piece →
  1. Semantic breadth

    shapes the knot's Radius

    • spread of meaning
    • range of topics
    • range of names and places
    • vocabulary variety
    • concrete–abstract mix
    • unrepeated lines
  2. Linguistic density

    shapes the knot's Tube

    • content words
    • noun-phrase length
    • action
    • concreteness
    • compression
    • capitals
    • punctuation
  3. Smoothness vs. punchiness

    shapes the knot's Tubular segments

    • smoothness
    • punchiness
    • one-syllable words
    • choppy lines
    • returning hooks
    • shared line endings
  4. Structure & rhythm

    shapes the knot's p and q

    • sentence depth
    • clause layering
    • variety of line endings
    • choppy lines
    • patterned hooks