Category Archives: Data Visualization

Comparative Methods for Image Analysis

(Image 1): Cover art for The Three Colors Trilogy: Blue, White, Red, by Krzysztof Kie?lowski.

The following are the notes of my presentation at the Art and Design Research Incubator (ADRI), Penn State, which took place on October 17, 2017.

You can download the tutorials of the workshop:

Navas_Data_Viz_IMG_Tut

AutomatorBatch_Compr

The presentation was designed to provide a clear basic example of how to perform comparative image analysis. For this reason, the following content is not an in-depth analysis of the three films. However, my long term interest on the color trilogy by Krzysztof Kie?lowski may find its way to one of my essays in the future.


Introduction:

The basic principle of image-mining is to be able to analyze large amounts of data (in this case a series of images) and be able to evaluate how they are organized with the aim to understand their differences and similarities. This approach is applicable to any large amount of images, which could consist of a series of photographs in an archive, a large amount of digital images in a database, such as Flickr, or a video or film. In all of these cases the methods for evaluating the construction of meaning relies in the analysis of particular patterns that may be at play in the body of information.

Image-mining is quite similiar to text-mining when approaching both with the purpose to evaluate how meaning is constructed based on certains types of patterns; however, because we understand images by comparing elements within a composition, and texts by sequential organization known as sentences and paragraphs, the way each of these forms is analyzed is different. I will go over text-mining in another occasion; For now, let us engage directly with image analysis.

Principles of Image Analysis

The basic approach to image analysis begins with the examination of a single image. Once we understand how this takes place, we can then implement this basic principle to a series of images. There is a limitation to this process when we reach thousands of images. At such point, we can implement big data analytics to be able to view trends that may not be possible to perceive with careful visual inspection.

In this occasion, as an example, I will analyze The Three Colors Trilogy: Blue, White, Red, by Krzysztof Kie?lowski. His work is optimal to analyze given that he is credited for being a methodical filmmaker who is keen on detail.

The best approach to image-mining is to work with sample data, meaning one can select part of the information for analysis. This is common in statistics, as is well known. In this occasion, I will focus on the opening scenes of Kie?lowski’s trilogy in order to move through the basic stages of analysis:

As noted in my first workshop presentation at ADRI, the process of data visualization consists of four parts (which can be subdivided into more parts of course):

  • Perform research based on a question (image, sound, text)
  • Gather data
  • Evaluate data
  • Re/present the data (image, sound, text)

 

In order to get a more complete analysis of the films, we would inevitably need to analyze sound as well, but as already stated, the image is the focus in this occasion.

My basic question is “What are the similarities and differences among the three

films by Kie?lowski?” To engage with this question, I performed the following method:

  • Choose subject (Trilogy Films) based on a question
  • Define parameters for sampling (around 60 seconds of Opening scene)
  • Evaluate (similarities/differences)
  • Open to further research based on initial results

 

Image 2: Cover art of Blue.

Let’s begin by considering the first minute of the three films (not including opening credits). Blue, starring Juliette Binoche, was released in 1993. It focuses on Julie de Courcy, who must cope with life after the death in a car accident of her child and husband (music composer Patrice de Courcy).

(View Blue, opening scene)

The first minute of the opening scene shows the closeup of a moving car tire, then cutting to the film credits and title, then back to the tire, then cut to the hand of a child holding a piece of paper outside the back passenger window, then cuts to the child viewing the road from the read window while yawning, suggesting a sense of boredom. One gets a sense of time passing as the light has changed from day to night.

Image 3: Grid-montage of Blue.

This scene when extracted shows a somewhat clockwork rhythm as is evident in the grid montage visualization. (image 3)

Image 4: YZ Slices of Blue.

Image 5: XZ Slices of Blue.

Such pattern can also be noticed in the YZ and XZ sliced visualizations. (Images 4 and 5) The point of this analysis is to compare Kie?lowski’s trilogy, so before we look further into these three images, let us view the other two films and their basic visualizations.

Image 6: Cover art of White.

White, starring Zbigniew Zamachowski, and Julie Delpy, is the story of Karol Karol, a Polish immigrant married to a French woman (Delpy). (image 6) The story is about Karol’s struggle to regain his sense of being byway of revenge, after his divorce, and loss of French citizenship.

(View White, opening scene)

The first minute of White‘s opening scene shows a large suitcase on an airport carousel, the camera follows the suitcase upclose while the film credits and title fade up. It then cuts to a low mid-shot of a man walking swiftly from left to right. The camera follows the man and when he stops to look to his right (the viewers left), the camera moves up to his upper body. This is Karol, the main character who appears concerned.

Image 7: Grid-montage of White.

This scene when extracted also shows a somewhat clockwork rhythm which is also evident in both the grid montage (image 7)  as well as the XZ and YZ slices (images 8 and 9).

Image 8: YZ slices of White.

Image 9: XZ Slices of White.

Let us look at the opening scene of the last film of the trilogy.

Image 10: Cover art of Red.

Red, starring Irène Jacob, is the story of Valentine Dussault, a college student who models part time. The film focuses on her relationship with a possesive boyfriend (who is only present on the phone), and a retired judge who owns a dog that runs astray, and is found by Valentine.

(View Red, opening scene)

The film opens with a close up of a phone. In the first minute, a man dials and we are immediately immersed in a tracking shot that takes us through all the phone cables across the ocean. In this case, the opening scene ends with the credits and title of the film.

Image 11: Grid-montage of Red.

While this opening may appear to be quite different from Blue’s and White’s, when viewing the grid montage  (image 11) and the XZ and YZ slices (images 12 and 13), we notice that there appears to be a similar rhythm, although this may not appear to be the case, given the fast movement of the camera through the ocean and landscapes.

Image 12: YZ slices of Red.

Image 13: YZ Slices of Red.

But what else can these images tell us? When we look at the three grid montage visualizations, (image 14) we notice that there are large sections with dark and light colors. They appear to be somewhat extreme in the visualizations, but this is not necessarily noticeable when viewing the actual films given that such changes are gradual. The overall colors of the films also appear to echo the title. Blue is predominantly blue, while White is quite dark, but certainly close to a whitesih/washed out greenish-gray value, and Red is predominantly red.

Image 14: Grid-montages of Blue, White, Red.

This is also confirmed with the XZ and YZ slices (images 15 and 16). However, a difference emerges when viewing these images. The film titles and credits appear at different moments. For Blue they briefly fade up around the middle, for White, they take much more of the opening shot, while for Red, they are at the end of the first sixty seconds. This difference may have taken place for many reasons, likely asthetic ones, which are unmeasurable. But what we can evaluate here is how these choices develop the uniqueness of the films, which are considered to be, by many critics, a complex body of work.

Image 15: XZ axis of the three films

Image 16: YZ axis of the three films

Running other analyses, we can learn a bit more about the structure of the trilogy. We can stack the images to evaluate its average intensity, median, standard deviation, and sum slices (images 17 – 19).

Image 17: Average Intensity, Median, Standard Deviation and Sum Slices of Blue.

Image 18: Average Intensity, Median, Standard Deviation and Sum Slices of White.

Image 19: Average Intensity, Median, Standard Deviation and Sum Slices of Red.

We can do this for all three films, and notice a few more variables (images 20 – 23).  For one thing, the film titles are quite pronounced, also, an object of focus emerges in all three. For Blue it is the car tire, for White it is the suitcase, and for Red, it is the phone.

Image 20: Average Intensity of Blue, White and Red

Image 21: Standard Deviation of Blue, White and Red

Image 22: Median of Blue, White and Red

Image 23: Sum Slices of Blue, White and Red

None of these objects would have stood out as they do in the visualizations, when viewing the actual film, and in this case, they appear as visual tropes that foreshadow the drama that is to evolve. Here we can notice that Kie?lowski’s methodical approach and attention to detail is confirmed.

Image 24: Datasheets of Blue, White and Red

But we can do more. We could also extract data from the images, and evaluate their mean, median, standard deviation to evaluate precisely how each of the three patterns may be similar and different (image 24).

Image 25: bar charts of Blue, White, Red

We can then take this data and visualize it as basic barcharts, that as expected, will also echo the pattern one perceives in the XZ and YZ visualizations (image 25).

Image 26: Mean of Blue, White and Red

Sometimes, in order to evaluate patterns, it is best to choose one measurement at a time (image 26). For instance if we take the mean, which is basically the average of color, and compare it directly with the corresponding slices, we may better understand how the pattern is constructed in terms of color values, and how this comes to inform our experience of the moving image (images 27 – 29).

Image 27: Comparison of mean to YZ slices, Blue

Image 28: Comparison of mean to YZ slices, White

Image 29: Comparison of mean to YZ slices, Red

We could take the time to match the numbers in the datasheets to the frames and see how much variation there is in the similarity of the patterns, but in this occasion, a visual evaluation already exposes Kie?lowski’s aesthetics, of having the tendency to make visual changes at about the same time during the introduction of the three films. In all three cases, there is a shift from dark to light. Adding the math, of course, can be useful for accuracy in a more detailed analysis that would try to evaluate extensively the director’s creative approach.

Image 30: Grid-montage of Blue‘s first 3 minutes

Image 31: Grid-montage of White‘s first 3 minutes

Image 32: Grid-montage of Red‘s first 3 minutes

The general take away from our analysis of the first sixty seconds of the films is that regardless of the elements that are part of the composition, all three films appear to juxtapose extreme lighting, which may not appear to be climactic in terms of the development of the story, but certainly function with the purpose to develop greater engagement with what is to come throughout the film. This relation of drastic light changes is even more evident if we also visualize the first three minutes of the films (images 30 – 32).

(Go over tutorial):

Navas_Data_Viz_IMG_Tut

AutomatorBatch_Compr

 

Data Visualization: Research and Creative Practice

 

Josh On, They rule: http://www.theyrule.net/

The following are the notes of my presentation, part of a workshop at ADRI on October 3, 2017. They are made available on Remix Data for those who attended the workshop, and for anyone online who may be interested in data visualization as a creative tool. The notes are meant to function as an entry point for anyone who is not familiar with data visualization, but hopefully, even those who are well acquainted will find relevant some of the concepts, and especially the examples discussed. You can download the technical tutorials of SVGs and a basic bar chart developed with D3, which were discussed at the end of the workshop:

Basic_SVGs_To_Share

Basic_Chart_To_Share

This session focuses on how data visualization relates to creative practice. First I will go over the basics of data visualization to then provide a few examples of artists, designers and researchers who have implemented some form of data visualization to produce creative works of art or design.

Three basic media forms to analyze:

Image
Sound
Text

No matter which type of form is analyzed, the data usually will  be organized as textual data that can then be visualized and represented in some way.

The data in turn can also be represented in the three forms

Image (data visualization)
Sound (sonification)
Text (datasheets)

There are many tools available for analysis at this point, for this reason one should become familiar with as many related to one’s interests before deciding to engage in advanced development.

Owen Mundy: https://owenmundy.com/

One thing to keep in mind is that information based production, which is the platform that informs our way of working at the moment, consists of key stages, which include:

  • An early stage in which the development took place by institutions and corporations that had major resources. This time goes back to the days when ENIAC was developed in 1945. In this case it was the military and a Research University (The Moore School at U Penn) that developed a room sized computer.
  • The next stage consisted of the privatization and commercialization of computing, leading to the culture of the 1960s closely linked with Xerox PARC in the Bay area of California.
  • The next stage would lead to principles of open source throughout the 1970s  which now inform our current culture. This stage is what made the Internet possible.
  • The attitude found in open source in effect is the backbone of contemporary collaborative production manifested in the arts and humanities as new media culture throughout the nineties to mid-2000s, which has now evolved in academia into the interdisciplinary field of the Digital Humanities.
  • Our current stage can be considered “Permanently Beta,” (a term I borrow from Gina Neff and David Stark). [1] The term points to the second of two ways of working:
    1) The first is using tools that are available, as one would use established software such as Photoshop, Illustrator, etc.
    2) The other is developing tools for one’s research that are shared with others for possible enhancement based on their own research interests. (It must be noted that the permanently beta principle is encouraged and promoted by corporations in order to get feedback as well as ongoing development based on a crowd sourcing model closely intertwined at this point with social media.)
  • In effect, the scholarship of the digital humanities is largely focused on how technology can be used for the evaluation and better understanding of cultural production.

One thing that I find Digital Humanists should be aware of is of becoming too focused on technological innovation. In effect, one may feel certain pressure, in order to become an established researcher, to develop something that appears technologically innovative as opposed to using tools that may already be available. This could lead to a model of innovation for innovation’s sake, which could be paralleled to moments in the history of art, in terms of “art for art’s sake.”

Brooke Singer: http://www.toxicsites.us/

The key thing, then, is to begin with questions that are of real interest to a researcher, and based on those questions, one should search for tools that will be useful. Eventually, one may develop a need for specialized tools which are not available, and that is when one may become an actual developer.

To drive the point home on this issue, in the spirit of remix, I borrow a quote from Charlie Gere, who in turn borrowed it from Gilles Deleuze: “the machine is always social before it is technical. There is always a social machine which selects or assigns the technical elements used.” [2]

If one has never had exposure to Digital Humanities methods, then one may not find how it can be relevant to one’s research. So, one should understand that the questions that may already be informing one’s research can be reshaped once one is exposed to different tools. With this in mind, I will explore some questions that I usually introduce to students who may not be familiar with text-mining or image-mining.

Key Questions for all analysis (image, sound, text):

  • How does the data help you evaluate your experience/understanding of the work?
  • Does your data analysis expose some elements of the work that you could not perceive by just viewing/reading/listening?

The above questions have embedded the foundational question for any subject: How does “it” work?

To engage with this basic question, one usually will analyze the material (image, sound or text) carefully, then writing down one’s observations, which in theory, others could review and analyze. With data-mining tools, one can actually use quantification to evaluate patterns, to then consider how such patterns may shed light on one’s question, which often times leads to more complex questions on the subject.

This process then consists of four parts:
• Perform research based on a question (image, sound, text)
• Gather data
• Evaluate data
• Represent the data (image, sound, text)

Giorgia Lupi and Stefanie Posavec, Data drawing pen pals, http://www.dear-data.com/

Evidently, this process cannot be covered in one session, so what I will show at this point are a few data visualization examples for image, sound and text, so that we get a sense how this process takes place. Given that this session is about data visualization, I will finish by focusing on the development of basic shapes that can be used to visualize any type of data. This might be basic for people acquainted with web development and programming, but the point is to keep in mind the conceptual process that informs how and why we come to choose certain forms to represent/visualize our research.

Artists and designers have actually used data visualization to produce creative work.  Some examples include (discussed with participants during live session):

Josh On, They rule: http://www.theyrule.net/
Owen Mundy: https://owenmundy.com/
Brooke Singer: http://www.toxicsites.us/

Image Visualization as a video presentation:
Josh Begley, Every NYT front page since 1852
https://vimeo.com/204951759
Cyrus Kiani
The Hawaiian Star
https://vimeo.com/37001373

Sound represented visually:
Manny Tan & Kyle McDonald,
Bird Sounds
https://experiments.withgoogle.com/ai/bird-sounds

Dear Data MoMA Collection:
https://medium.com/@giorgialupi/dear-data-has-been-acquired-by-moma-but-this-isnt-what-we-are-most-excited-about-bdaa3376d9db
Entry explaining the process:
Data drawing pen pals
http://flowingdata.com/2015/03/19/data-drawing-pen-pals/
Actual Project
http://www.dear-data.com/

In the spirit of the above examples, particularly the last one, I will focus on basics, because, as you can see, we don’t have to program to begin visualizing information. We can draw our interpretation of data. It is, however, important to eventually program your visualization so that it can reach a fuller potential. What is worth noting here is that when we visualize data, we are actually interpreting it. Data visualization can be a fair presentation of raw data when we consider how and why we choose colors, line shapes, spacing between objects etc. Such basic graphic objects  affect how the data is perceived. We must be conscious of this aspect of data visualization, and make sure we are being fair to the project when we decide to use specific visual elements to present factual information.

Data Visualization is in large part developed with SVGs. For this reason, I will go over how basic SVG shapes are developed. It is my experience that having a basic hands on understanding of the foundation of tools one will use helps one develop an intimacy that otherwise will never emerge if we only function as users. In other words, knowing what the code behind tools actually does helps in deciding how to use such tools.

You can download the tutorial if you don’t know how to use SVGs by clicking the links below. A basic bar chart is also included which can help in developing an initial sense of how data viz functions.

Basic_SVGs_To_Share

Basic_Chart_To_Share

footnotes:

[1] Gina Neff and David Stark, http://www.umass.edu/digitalcenter/events/2002Workshop/Papers/stark_permanently.beta.pdf, accessed October 2, 2017

[2] Charlie Gere, Digital Culture, “Introduction,” Digital Culture (London: Reaktion Books, 2008), 17.