To make a long story short, I didn’t get much useful out of this exercise.
This morning in one of the digital humanities committees I’m on at work we were discussing chat bots and AI, the ethics and legality of using texts generated by these types of software. There is a lot of research that needs to be done to catch up on (and keep up with) these technologies as they develop. But one of the points we came upon several times was the accuracy of the results.
In one instance, the faculty member asked the same question in a number of different ways and got back the same answer — the wrong answer — every time. In another she was asking about a specific work by a specific author, but every answer she got was about a different work by that author — the author’s most-popular work. But it knew nothing about that author’s other work.
So I decided to put it to work on something that comes up a few times a year, placeholder text for program notes. Similar to the DALL-E images I had generated for similar kinds of posts. Only this time, we had to have program notes that were factual.
Mozart
My first attempt was to get program notes for Mozart’s “Haffner” symphony. I should give credit to the AI for understanding the concept of program notes for a piece of classical music. It got the composer’s full name, the K. number, the key, date of composition, why and for whom it was composed all correct. It understood the structure of program notes as they are generally presented.
My prompt was simply:
Can you write program notes for Mozart’s “Haffner” Symphony?
Sure! Here are program notes for Mozart’s Symphony No. 35 in D Major, K. 385, also known as the “Haffner” Symphony:
Composer: Wolfgang Amadeus Mozart (1756-1791)
Title: Symphony No. 35 in D Major, K. 385, “Haffner” Symphony
Date of Composition: 1782
So far so good! Then it continues:
Wolfgang Amadeus Mozart wrote his “Haffner” Symphony in 1782 for a performance in Salzburg, Austria to celebrate the investiture of Sigmund Haffner the Younger as a knight. The symphony was commissioned by the Haffner family and was one of the several works Mozart wrote for them.
Again, pretty good. I don’t know what the “several” works are, though I do know the Haffner Serenade. Then it continues with a perfunctory boilerplate analysis of the symphony.
The “Haffner” Symphony is a buoyant and lively work that showcases Mozart’s mastery of the classical style. The first movement, marked Allegro con spirito, is characterized by its bright and energetic theme, with a lively exchange between the strings and woodwinds. The second movement, Andante, is a graceful and lyrical contrast to the first, featuring a beautifully melodic theme that is passed between the different sections of the orchestra. The third movement, Minuet and Trio, is a charming dance in triple meter, with a trio section featuring solos by the oboe and bassoon. The final movement, Presto, is a lively rondo that brings the symphony to a rousing conclusion.
A couple of quibbles. Looking over the score, in the Andante there is no theme passed between different sections. The violins have it, period. And in the third movement the “solos by the oboe and bassoon” are just two-note motifs, first in the oboe, then the bassoon, and that’s it. Everything else in it could be used to describe any of Mozart’s major-key four-movement symphonies. A point to revisit.
Vivaldi. Did you even listen?
The other query I posed was:
Write program notes for Vivaldi’s Concerto “Alla Rustica” in G Major RV 151
That’s unambiguous. IMSLP has it as RV 151, “All Rustica,” in G Major. Here’s the first page of the score:
And this is what Open AI has to say about it:
The first movement, marked Allegro, opens with a lively and energetic solo violin passage that sets the tone for the rest of the piece. The solo violin and strings engage in a lively dialogue, with the solo violin taking the lead and the strings providing lively and energetic support.
Now what are we looking at. First, it’s not marked Allegro, it’s marked Presto. Second, there is no solo violin passage. In fact, the entire first movement is pretty much the firsts and seconds playing in thirds. There’s no dialog. The texture is the same throughout.
Well what about the second movement? Open AI says:
The second movement, Largo, is a beautiful and lyrical contrast to the first, with a haunting and expressive solo violin melody.
Here’s the entirety of the second movement:
It’s not Largo, it’s Adagio. There’s no lyrical element at all, only an overture-style dotted rhythm. Now it’s quite possible that what Open AI “listened” to was a rendition of this as a solo violin cadenza, in the manner of the second movement of the Third Brandenburg Concerto (only longer).
At this point I’m going to check my sanity and see what performances of this are available online. Indeed this one does exactly that treatment — but note that the solo is the soloists improvisation (and very nice too), not Vivaldi’s.
You can search YouTube for these (“vivaldi alla rustica rv 151”).
The third movement is more of the same, I’ll only mention it that Open AI says it’s Presto, but it’s actually Allegro.
As with the Mozart example, boilerplate language that can serve to describe many of Vivaldi’s concertos — but explicitly not this one, which is a concerto gross, not a violin concerto.
Other notes
Over on theatlantic.com, Charlie Warzel notes:
ChatGPT’s current model is already infamous for confidently stating false information. Yesterday, Reuters reported that one of Bard’s demo answers, which concerned space telescopes, included a factual inaccuracy.
…
Already, Google is facing financial repercussions for its Bard presentation: The report of Bard’s factual error caused the company’s stock to slide as much as 9 percent. It also led to arguments over whether Bard was actually wrong. The Financial Times wrote that the answer was only misinterpreted, whereas an astrophysicist insisted that the error was clear and factual. This confusion is a glimpse into our immediate AI future, one in which humans disagree about whether the machines are telling the truth, while fortunes are gained and lost in the process.
The Vivaldi program notes I received were pure bullshit, and while grammatically and formally exactly like program notes you would read in a real concert they had nothing to do with the actual music you’d be listening to.
OpenAI released ChatGPT, which has upended many conceptions of how machines can interact with humans, passing graduate-school exams, flooding the internet with confident bullshit, writing news articles, and helping people get jobs and cheat on tests.
Given all of this it’s hard to believe that undergrads, let alone grads, would be willing to rely for long on such an undependable source of information. After all, we all knew the guy (it was always a guy) in college who pretended to know everything. Is that who you’d outsource your term paper to?
I asked Open AI what happened, and it apologized. Really.
So then this how it learns. Happy to do my part. I’ll be issuing an invoice shortly!
Where the answer is leading
The solution to this is going to lie in a couple of different places. Use the iterative nature of the chat bot to refine your text. Do this by turning on your bullshit detector and either correct the errors, or tell it to skip over areas that it doesn’t know anything about. For example, it’s pretty clear that it can’t read a musical score yet.
Then you’re going to have to do your own fact checking. For the purposes of program notes we can see that it knows this goes there, and if it doesn’t have an authoritative this it will make something up based on a rough average of what it knows from similar cases. Likewise, tell it to bring up topics that you think it can handle well. I put in a second request:
Antonio Vivaldi was an Italian Baroque composer and violinist, who lived and worked in Venice during the early 18th century. The Concerto Rustica was likely composed during the middle period of Vivaldi’s career, around the late 1710s to early 1720s, when he was at the height of his fame and success as a composer and performer. At this time, Vivaldi was working as a composer, violinist, and teacher in Venice, where he was active in the city’s vibrant musical life.
The Concerto Rustica is one of Vivaldi’s many concertos and is characterized by its rustic or folk-like melodies and themes. This type of folk-inspired music was popular in Italy during the Baroque period, and it was not uncommon for composers to incorporate elements of folk music into their works. The specific folk influences on the Concerto Rustica are not well documented, but it is likely that Vivaldi drew on the traditional music of the rural regions surrounding Venice for inspiration.
As for its influence on future works, it is difficult to say how directly the Concerto Rustica influenced other composers. However, Vivaldi’s music, including the Concerto Rustica, was highly popular and widely performed during his lifetime and continued to be so in the years following his death. His music, along with that of other composers of the Baroque era, would have certainly had an impact on the development of classical music and the classical concerto form.
Overall, the Concerto Rustica is an important work in Vivaldi’s larger body of concertos and provides a glimpse into his versatility as a composer and his ability to evoke different moods and themes through his music.
For the purposes of general-audience program notes this isn’t bad. I’d want to fact check a few items before publishing it, but as a first draft it’s a big time saver. Combined with some of the earlier iteration, such as:
Composer: Antonio Vivaldi (1678-1741)
Title: Concerto “Alla Rustica” in G Major, RV 151
Date of Composition: Early 18th CenturyAntonio Vivaldi was one of the most influential composers of the Baroque era, known for his virtuosic instrumental works and his innovative approach to musical form. The Concerto “Alla Rustica” in G Major, RV 151 is a delightful example of his work…
In another column on theatlantic.com, Charlie Warzel claims that “talking to AI” is going to become one of the more important skills we can learn in the 21st century. Likens engineering successful prompts to
… managing a bright and eager junior employee … enthusiastic and skilled, but also inexperienced and thus more likely to make subtle, but crucial, mistakes. It’s also great at bullshitting when it doesn’t have the answer.
So I guess it’s like telling said junior employee, “great, now take it back for another version. Correct points 1, 2, and 3. Write with a different tone. Add sections on these key points.”
And at that point, you’re getting somewhere.
Fact-checking is a useful skill. We’ll need to learn it more than ever.