Category Archives: Video and Image Analysis

Analysis of 30 YouTube Music Video Mashups

30MashGridWeb

Figure 1: Visualization of 30 Music Video mashups taken from YouTube. Mashups were uploaded between 2006 and 2011

I am currently finalizing research on 30 YouTube music video mashups, uploaded between 2006 and 2011. I started this research during the last days of my post-doctoral work at the University of Bergen in May of 2012. I was able to refocus on this project during 2015 thanks to the support of a Faculty Research Grant from The College of Arts and Architecture at The Pennsylvania State University. I thank them for their ongoing support. In what follows I will introduce the principles of this research project.

10PrelemViz

Figure 2: Preliminary Visualization of the first 10 of 30 music video mashups. The list includes from top-left to bottom-right: 1) Danger Mouse Encore (Grey Album, Jay-Z and Beatles), 2) Radiohead vs. Dave Brubeck,  3) 99 Problems with Buddy Holly, 4) Ray of Gob (Madonna vs. Sex Pistols) , 5) The Strokes vs. Christina Aguilera, 6) Eminem Featuring Aerosmith Sing On For The Dream Moment, 7) Blondie Vs. The Doors – Rapture Riders, 8) Boulevard Of Broken Songs – Green Day ; Oasis ft.Travis ; Eminem, 9) Sour Glass (Portishead vs. Blondie vs. Kanye West), 10) Weezer vs. Queen.
Color code: light-green = special effects or titles, blue = main-track/instrumental, yellow= lyrics/complementary track, red = footage of both videos combined.

During my postdoctoral research I worked on three case studies of YouTube video memes. I published my findings in the paper “Modular complexity and remix: the collapse of time and space into search” Based on the editing patterns that I noticed the YouTube video memes shared,  it became evident to me that a particular grammar and syntax was at play in the way people edited diverse forms of music video remixes; this grammar also appears to be linked to the way music mashups are produced and function, and for this reason, after I finished my research on the music-video memes, I decided to look further into video-music mashups. I chose 30 of the most popular mashups between 2006 and 2011.  Given that remix as a form of cultural production is closely linked to basic music remixes it is worth evaluating how meme patterns move across different genres that combine image and sound as a single form of expression. Music video mashups are good case studies for this reason.

To analyze the patterns of the memes more accurately, I realized that it was not enough to visualize a grid of the video footage as I had done for my previous research. I needed to see how the footage of different videos was edited as one to complement the actual music remix. Therefore, I developed a color-coding system to mark the video edits. I did the first 10 manually, by laying color over the video montage (Figure 2).

This process took too long and was not very accurate, so I developed a batching method in which I evaluate the frames that correspond with the original music videos and assign them a numbered sequence that can then be run to create a montage that matches the original video montage. I share the examples of the first three videos below.

1_DangMousMontWeb

Figure 3: “Danger Mouse Encore (Grey Album, Jay-Z and Beatles)” Grid-montage of music video mashup. Video available at http://www.youtube.com/watch?v=qFd1QvNPKmI

1_DngMsJayZColorWeb

Figure 4:  “Danger Mouse Encore (Grey Album, Jay-Z and Beatles)” Video color-coded pattern. Color code: yellow = special effects or titles, blue =  instrumental/main-track, green = lyrics/complementary track, red = footage of both videos combined–or hybrid video with special effects.

2_RadioBrubeckMontWeb

Figure 5: “Radiohead vs. Dave Brubeck” Grid-montage of music video mashup. This version is no longer available on YouTube, but an alternate remix can be found here: https://www.youtube.com/watch?v=0PQQPUb8Rl0

2_RhBrubkColorWeb

Figure 6:   “Radiohead vs. Dave BrubeckVideo color-coded pattern. Color code: yellow = special effects or titles, blue =  instrumental/main-track, green = lyrics/complementary track, red = footage of both videos combined–or hybrid video with special effects.

3_JayBuddyMontWeb

Figure 7: “99 Problems with Buddy Holly” Grid-montage of music video mashup. Video available at: https://www.youtube.com/watch?v=4bUAzC9wgkw

3_JayZWeezColorWeb

Figure 8:   “99 Problems with Buddy HollyVideo color-coded pattern. Color code: yellow = special effects or titles, blue =  instrumental/main-track, green = lyrics/complementary track, red = footage of both videos combined–or hybrid video with special effects.

Based on these three videos we can see that the approaches vary but nevertheless there are some similarities already evident. For one thing, there is a need to go back and forth between footage in order to reinforce the fact that the video is of a music mashup of two songs. The first two videos show some aspect of the video (red) that supports the aural experience of the sound mix colliding. This is also evident in my preliminary visualization of the first ten videos (figure 2). There is an exception of two videos (mashups 3 and 5), which don’t have any combined or hybrid footage (red), but only juxtapose material of the originating sources.  When listening to the actual music remix, it becomes evident that it is because of the way the two songs are mashed that the videos are edited with basic montage. In all of the videos I have analyzed thus far, the image appears to support as best it can the aural experience of the remix. The reasons behind this begs a detailed theoretical examination. It is my aim that my analysis will provide a good sense of how image and sound complement each other in music video mashups. I have looked at more recent mashups on YouTube since 2012, and find that the elements in the 30 videos I am analyzing are still at play. This may mean that certain aesthetic and expressive conventions of communication are already well in place among video and music remixers. In the future, I will be sharing more details of my findings on all 30 videos in a formal paper.

Analysis of the Films In Cold Blood, Capote, and their Corresponding Novel and Biography

InColdBloodCapote

Figure 1: selected shots from Capote (left) and In Cold Blood (right).

Interdisciplinary Digital Media Studio is a class in the IDS program in The School of Visual Arts (SoVA) at Penn State in which students are introduced to methodologies and conceptual approaches of media design. For the class, I taught them how to research and develop design presentations with the implementation of data analytics for moving images and texts.

One of the assignments consisted in analyzing the films Capote (2005) directed by Bennett Miller and In Cold Blood (1967) directed by Richard Brooks in relation to their corresponding books, Capote by Gerald Clarke and In Cold Blood by Truman Capote. We viewed the films in class, and read, both, the novel and the biography. The class then analyzed the respective books by doing word searches, analysis of specific passages, and creative approaches by the respective authors, to then evaluate those searches in relation to the films.  For the films I provided montage visualizations, which are selected screen shots representative of all the scenes (figures 2 and 3).

InColdBloodSmall

Figure 2: Visualization of the film In Cold Blood (1967). Click for a larger image.

The students were free to use the film visualizations and the data-mining of the texts according to their interests. Some of the options I suggested, but which were certainly not the only possibilities, included: times names or any other terms were mentioned, difference in the way particular moments appeared in the films and the books, and which parts were  omitted.

CapoteMontSmall

Figure 3: Visualization of the film Capote (2005). Click for a larger image.

The students had to design an infographic, or a similar form of visual design that presented their findings in an easy to understand format. The students had to also write an assessment of their findings. What follows are selected projects which I consider successful for elements that I explain for each case. All my students worked very hard on their visualizations, and ideally, I would like to show all of their projects,  but for the purpose of this post, I selected designs that show  unexpected or experimental approaches in the implementation of analytics to find possible answers to diverse questions on the same subject. The samples that follow are successful in some ways, while having shortcomings in others.

M_Regan_Capote

Figure 4: Visualization of common elements between Capote, the book and its respective film adaptation, by Michael Regan. Click image for larger file.

Michael Regan looked at commonalities and differences in Capote the book and film. In this regard he writes:

The focus and content of my infographic is in the arrangement of common elements between a book and its film adaptation. This is a very relevant way of viewing the comparison of two media representations of the same content. It is very interesting to view how differently the film creators had to move content to best adapt the form. The In Cold Blood film rearranges everything from the book into a different order to better fit a film format, while the Capote film takes most of its content entirely from one section of the book. These are two ways of adapting a book into a film, and looking at the infographics allows a very quickly visual way of understanding these techniques. It also shows the way that the books are constructed in the first place. The In Cold Blood book shifts back and forth often among the murderers, the victims and townspeople of Holcomb, and the investigation. This helps to explain the switching back and forth of elements and their ordering between the film and the book. This is contrasted by the writing style of the Capote biography. The small relative amount of time the book spends on the In Cold Blood writing process, as shown by the infographic, shows the focus of Gerald Clarke, the author. He chooses to explore the entirety of Capote’s life, without giving an undue amount of attention or priority to the In Cold Blood part of his life, however sensational a time it may have been.

Regan’s visualization represents the emphasis of the film on particular aspects of the book, and where they may or may not overlap. One can get a decent sense of the adaptation process, and to some degree, assess how such process is enacted in the particular production.   His visualization is indexical, meaning that it allows the user to go to specific areas that Regan evaluates are important to the process of adaption from book to film.

AndrewHeoAdjusted

Figure 5: Visualization of color mentioned in the two films and books analyzed in class by Drew Heo. Click image for larger file.

Drew Heo decided to focus on the role color played in the films and the books. In his assessment paper he writes:

Color is a visual component, and Truman’s novel and biography are black and white on paper. Due to the two differing mediums, it’s only fair to allow both works to shine in the fields they are best in. The bar graph is a visual indicator of the specific mentions of colors in the texts of both books, as well as the amount of times a color is verbally spoken in the films. As expected, the films fall short when compared to the writings, as color is a visual thing and not used as much in speech. In order to compensate for the lack of mentions, below the graph is a condensed form of the colors present in both films in a “Movie Barcode” format, which has become recently popular on the internet for analyzing films.

To consider how color plays a role in the film is a unique question certainly worth pursuing. The accuracy of the numbers do need to be double checked, but what is worth noting here is his particular approach to evaluate a concept across media that is not easy to quantify. If anything, his focus and approach demonstrate a potential for more in-depth analysis of details shared across works that are adapted and crossover relevant productions that are often compared, but hardly ever analyzed quantitatively.

EthanInfoGramEdited

Figure 6:  Recursion of characters in the biography, novel, and respective film adaptions, by Ethan Jones. Click image for larger file.

Ethan Jones focused on the number of times the names of particular characters appeared in the text, and the number of times each character appeared in the film.  He writes:

There was a decision early on to reduce the commonalities between all four works by omitting what was exclusive to one story or the other. For example, the Truman Capote character is omitted because of his lack of presence in the In Cold Blood story (book and film). On the other side of that, minor characters from In Cold Blood (book and film) were discarded because of their lack of involvement in the Capote biography. Several characters remain constant, such as Perry, Dick, Alvin Dewey, Susan Kidwell, and the Clutter family. Simply looking upon the amount of times a given character was mentioned proved to be sufficient.

With Jones’s barchart we can evaluate how the characters are at play in all four works. One thing to note is that he counts the appearances of a character in the respective films every 5 frames. This, however, is dependent on the selective shots I provided, so the number of times a character may actually appear is not related to every five frames from the actual film but frames based on my selection  of shots representing each scene. Nevertheless, the chart does provide a general idea of the prevalence of the characters in the respective publications and films. This can be cross-examined with the analysis of other students which is not included in this entry.

Nikki_in_cold_blood

Figure 7: Analysis of parts of the book and film where dialogue did not play a major role, by Nikki Tatsumi. Click image for larger file.

Tatsumi focuses on parts that had next to no dialogue in both the film and the book, In Cold Blood, she writes:

For this project I focused on the parts of the movies and books where dialogue did not play a major part in the creation of the scene. This was most relevant in the book In Cold Blood. This book, borderlining a research novel, was beyond descriptive. Its specificity allowed the reader to truly identify and imagine these people to the point of skin color, body language, tattoos and countless other tidbits that Capote decided to include. If not enough, Capote’s rendition of each character’s backstory also helps to fuel the reader’s imagination. It is through these scenes that I think Capote is the most successful. Though dialogue gives a character a voice, Capote was able to utilize his research and observations to create characters who did not need one. By their mannerisms alone, Capote could write this whole story. But when these heavily described sections of the book meld with the movie. Directors used the score to help make up for lost information through the text. In Cold Blood and its movie, had
the best transition.

In the movie version, heavy use of jazz influenced music helped to create the mood. It was a lot more foreboding due to the the choice of score. It was the type of music that was not soothing or collected, it was erratic and soft at times, loud and uneasy at others, as if the viewer did not expect what was next. The scenes that this score was utilized really well were the juxtaposed part of Perry’s past and the beginning scene with Perry at the bus terminal. They utilize the uneasy jazz to mirror Perry’s
feelings at both times.

In Tatsumi’s design we can note that she was selective of which parts to feature, as there were many more moments and passages in both the book and the film that would fit the subject of her research. In this case, we can consider her focus and design similarly to Heo’s, with potential for more in-depth exploration. Her focus also shows how analytics, while at times may be presented to be objective in what the results may be, are actually informed with particular questions and interests that drive the research. This is something that individuals using quantitative tools need to be more honest about, and not claim that what they found is entirely “objective.” Tatsumi’s focus is unique in that she focuses on moments in which dialogue did not play a major role. No other student considered this, and I never would have thought of this focus. In this sense her research points to an aspect of the film that could be very interesting to further analyze.

One thing that I should note is that her design could be much better. However, given that many of the students were dealing with multiple elements of creativity and research I did not push her to redesign her presentation given that she had touched on an interesting aspect of both the film and the book that she could have explore more in-depth.

 
MeganColdBlood_Capote

Figure 8: Diagram of biography, novel and both films by Megan Coren. She focused on similarities of the four works. Click image for larger file.

Megan Koren focused on the similarities of all four works. Similarly to Tatsumi, her diagram shows that she was rather selective. She writes:

The nature of my infographic does not show many of the differences between the books and films; rather, it shows the similarities of In Cold Blood and a follow-up comparison of how Capote visually paralleled this information. Because Capote was a secondary addition, I feel I provided a skewed representation of the biography. I included text from the book that aids the comparisons in my infographic, but Capote is not really about the novel In Cold Blood.  It is about Truman Capote’s life, spanning childhood through adulthood.  Although interviews with the Clutters cover only roughly 50 pages of the monstrous 547-page biography, the film diminishes this biographical time-line and completely focuses on Capote’s writing of the book.  The film concentrates on the obsessive Truman himself, while shot-wise it accurately parallels the In Cold Blood film. We see the similarities between Capote and Perry and why Capote sympathized with him so strongly. Regardless of their different perspectives, I was genuinely impressed at how many connections the directors were able to make between scenes in the Capote and In Cold Blood movies.

This particular presentation is easy to read, and one is able to figure out how the four works  are related according to Koren’s interest.  However, it does have inaccuracies that are evident upon closer observation. for instance, the moment when Perry speaks on the public phone is not at 0 minutes of the film, but actually more or less around 10 minutes into the film. Koren should have spent more time with the films to evaluate their respective chronologies. If she did this her assessment, which is quoted above would be stronger.  Her selection of passages from the books also lack a systematic approach, which makes one wonder why the citations were chosen over any others. In any case, Koren’s design shows potential in exploring formal aspects of presentation of a large amount of material. She needed to spend more time working on the accuracy of the data and information–her design, conceptually, shows potential, although it could be further polished for a more elegant read.

 
Yoder_final_v3-04

Figure 9: Visual analysis by Caleb Yoder of the names “Dick” and “Perry” in book publications Capote and In Cold Blood, and their respective film adaptations. Click image for larger file.

Caleb Yoder decided to be very specific in his analysis and focused just on the names of the two killers.  He writes:

After data-mining information for Dick and Perry, I found that the books are often able to provide a lot more in terms of description for the characters. On page 325 of Capote, Clarke is able to give a thorough description of Perry’s appearance, personality, and childhood. The films need to rely on the actions of the characters or their conversations, and in some cases even flashbacks (like Perry’s multiple flashbacks to his childhood in the film version of In Cold Blood), in order to convey this information that the book lists off so easily. Film must also take some creative liberties with how the story functions visually–the book doesn’t describe every set piece, and so the film inevitably conveys some information independent of the book.

Both mediums make the most of their capabilities, though. The book In Cold Blood familiarizes us with the Clutter family in a way that the film fails to do (most likely due to time constraints). Our greater intimacy with the Clutter family and characters like Bobby Rupp in the book serves to make the Clutters’ deaths and the impact they have on the Holcomb community more salient to us. The film is also able to stir emotional responses through its use of music, composition, and the pacing of shots. A great example of this appears in both films when Perry is hanged at the end. The audio swells, Perry breathes heavily, and the intensity increases until the trapdoor opens abruptly.

Yoder’s is one of the most accurate analysis submitted. His focus on only two terms allowed him to develop a detailed visualization of the films and books. In turn, he is able to provide specific pages in the books and times in the films when particular moments take place, in which both Dick and Perry are prominent. This gives us a strong sense of how the stories of Dick, Perry, and Capote were intertwined in real life as well as in the semi-journalistic fiction created by Capote himself.

The visualizations presented here should be considered starting points for the future of analysis by designers. The students in my class eventually applied the methods and critical skills developed in this particular analysis to develop a research & design project of their own, which was the final assignment of class.

There is much to be said about each of the analyses covered here, as well as others. For instance, we can see how by using analytic tools with quantitative capabilities we are able to dissect the grammar of visual and textual languages to better understand their intersections. All of the projects in class, including those not featured here did show a decent balance in quantitative and qualitative analysis.  The challenge for a few of the students as is evident in some of the examples part of this entry is to push further for accuracy in their results. This is a skill students keep working on as they move on with their studies.

Timeline of Pulp Fiction: Actual Version and Chronological Edit

PulpThumbOf4

Figure 1: four shots from around a third into the film. Left is original edit, Right is chronological edit

During the Fall of 2013, I analyzed Pulp Fiction with my students in my Video Art Class for the School of Visual Arts at Penn State. One of their assignments was to produce  a video and then re-edit it to tell the same story but in different order, and therefore explore how aesthetics play a role in experiencing a narrative.   We went over a few examples that would give them ideas, some of the links I provided as resources included Pulp Fiction and Memento. I share them below:

Infographic of Pulp Fiction in Chronological order:
http://visual.ly/pulp-fiction-chronological-order

What Watching ‘Memento’ in Chronological Order Can Teach About Story Structure:
http://nofilmschool.com/2013/06/watch-memento-in-chronological-order-story-structure/

Timeline for Memento:
http://www.thehighdefinite.com/2012/02/the-timeline-for-memento/

Memento Chronological Order (make sure to watch the original film before viewing this):
http://www.youtube.com/watch?v=Ishi0TyiBrs

We also viewed a chronological version of Pulp Fiction which was available on line but, unfortunately, was taken down. And I also presented in class a two column set of still frames of the two versions of the film (figure 2) of the way it was edited by Tarantino (left), and the chronological order version (right). We discussed how the film has a particular open-endedness  due to the fact that its beginning and ending  appear to be the middle of the story. This is fairly well known but it  becomes more than evident in the two column visualization I provide below that the editing of the film is not as simple:

PulpOrigReEdit900

Figure 2: Pulp Fiction, original on left, chronological on right.

To generalize that the ending is the middle is a misconception, once we take a careful look at the visualization above, because we can notice that  both versions of the film are actually almost in the same order for the first 10 minutes more or less. Notice that the early crime scene in the apartment happens almost at the same time in both versions. It is when Wallace’s wife is introduced that real changes can be noticed. But this is a bit difficult to grasp  because the two opening scenes for the original and chronological versions provide diverging intertextual framings to engage with the characters as they are introduced throughout the film: to view a scene at a coffee shop and a scene of a boy and a Captain offer contrasting contexts for the next scene of Jules and Vincent.

One can argue that these scenes are different because in the chronological version the scene with Captain Koons is taken out of the original order in which the film was edited. This is important to note because aside from this scene, all that Tarantino appears to have done when he edited the film to have the middle  as its beginning and ending is to develop a conventional story that is told chronologically. It is Captain Koons’s scene, then, that appears to stand out in the chronological order, because it functions as a flashback of Butch’s early childhood–notice that  in the original version of the film the very next scene is Butch waking up from a dream before his boxing match (hence linking him to the flashback).

To stay true to a chronological timeline the scene with Koons  has  to be moved to the beginning of the film because this moment happened much earlier than all the other events.  And this makes Koon’s scene the only exception to an otherwise minimal shift in the middle becoming both the beginning and ending of the film. But the flashback could be considered part of the chronological order because it is really Butch who is reliving something in his dream, and this reliving can also be considered part of his present. This then complicates the basic premise of the middle of the film being the ending and beginning of an otherwise chronological edit. What follows shows how the film would appear if we took the now conventional notion that the middle is the beginning and end:

PulpOrigReEdit900RG

Figure 3: Pulp Fiction, original on left, chronological on right. Sequences visualized to trace the chronological ending of the story.

When we look at this visualization we notice that the chronological ending of the film does not fall in the middle of the original film as is commonly argued. This means that the story is further edited. When breaking it down in more detail with color-codes, we can notice the following differences:
PulpOrigReEdit900RedGreenEdits

Figure 4: Pulp Fiction, original on left, chronological on right. Actual editing of chronological sequences  for both original and chronological versions.

We can see that the original film does not fully follow a chronological order that was simply edited to make the middle of the chronology the beginning and ending. The chronological ending of the film takes place about two thirds of the way into the film, while the ending of the original does fall more or less around the middle of the chronological version. But even when this happens we can notice that parts of the chronology are moved around to enhance the experience of the story. For instance, the opening of the original takes place just before we reach the middle of the film, meaning that this part of the story is part of the ending, of course.  We can look at each of the other segments and notice have they are shifted to tell the story in a way that will be more interesting than it being simply chronological.

If we number the order of the original edit and juxtapose it with the chronological version, we get this:

1 | 4
2 | 2
3 | 6
4 | 1 & 6
5 | 3
6 | 5

I numbered 1- 6 the chronological sequences of the original version (left column), and repositioned them into the chronological version of the film (right column). We can notice that the two opening scenes are different (diner and Captain Koons), but the very next scene is the same (Jules and Vincent in the car on their way to do a job). The second sequence in the original version is then split in order to turn it into the final chronological sequence (6,  Jules and Vincent finishing the job to end up back in the diner), this is why 3 and 6 match about a third of the way into both film versions. Notice that six  then comes together with sequence 1 to end at the very middle of the chronological version and match sequence 4 (Vincent and Mrs Wallace) in the original edit. It is sequence 5 (Butch fighting, escaping, running into Vincent, and Mr. Wallace,  confronting the gimp, and the eventual get away) that is the actual chronological ending of the story, but we see that in the original edit this one is followed by section 6, which is the scene of Jules and Vincent at the diner, this is also sequence 1, as we know.

So, to say that Tarantino merely took the chronological development of the story and split it for the middle to be the beginning and ending is really not correct–this is what appears to be commonly assumed by some people when they think of the middle of the story being the ending and beginning of the actual film. There is  much editing at play which makes this film more complex formally speaking.

But the editing is not so radical because going back and forth between closely related timelines is quite common in films. What is peculiar of Pulp Fiction and some other films by Tarantino is that they are edited as though things are happening now, there is no clear hint for the audience to acknowledge that we are going back and forth in time.  Viewers must acknowledge this as they try to make sense of the story. This approach by Tarantino challenges the cinematic aspect, because the audience must remind themselves that they are viewing a film and they must make sense of its sequences. The audience must try to make them fit so that the story comes together; but this must take place in the viewers’ minds. Pulp Fiction, arguably, is a reflective exploration of how we come to engage with films, how the process of editing can become a form of communication that also questions how we try to make meaning of the content being experienced.

Another challenge of the film is the cultural stereotypes it presents, and how viewers must question them as well. This aspect of the film is more open ended, and Tarantino has been accused of promoting certain stereotypes, particularly of African Americans. Those questions are very important to discuss, and must be.  And I take the time to discuss them in actual class. For this post, however, I focused on the formal aspects of Pulp Fiction.

The Beginnings of Remix Data as Research: Three Case Studies On YouTube Video Mashups

The beginnings of this site, remix data, in terms of my research goes back to my analysis of YouTube Memes. Below I repost information that is available on Remix Theory, which eventually led to my current focus on remix and data:

Three case studies: The Charleston Style meme, Radiohead Parodies, Downfall Parodies

Research on Remix and Cultural Analytics

Post-doctoral research for The Department of Information Science and Media Studies at the University of Bergen, Norway. For this 2012 research I implemented cultural analytics techniques to analyze YouTube video remixes.  My research was performed in collaboration with the Software Studies Lab at the University of California, San Diego. There is a second part of this research which I’m currently developing in collaboration with the New Media and Interdisciplinary Digital Studio Program in the School of Visual Arts at Penn State.  Below are links to brief descriptions of already published material.

Part 1: Charleston Style

Part 2: Lotus Flower Parodies

Part 3: Downfall Parodies

Part 4: Sliced Images of Three Case Studies

Part 5: Vroom at Calit2

Online research material used for the three case studies

Peer review publication, Anthrovision, 2012
Modular complexity and remix: the collapse of time and space into search