Josh Pollock

I’m a fifth year PhD student at MIT. I work on domain-specific languages for graphic representations.

I like doing research. My research so far has been in programming languages (PL), human-computer interaction (HCI), and data visualization (VIS). I especially like building systems that formalize, clarify, and operationalize research ideas.

The activities that excite me tend to have these three qualities:

postrational

Integrates formal (explicit, technical, abstract) and intuitive (implicit, experiential, concrete) ways of being and doing. Integrated approaches use direct observation to develop just the right amount of structure, which in turn enhances direct observation.

interbeing

Prefers clusters over categories. Identifies larger spaces or fabrics where clusters live. Rather than seeking to separate phenomena, activities that embody interbeing seek to understand how phenomena are connected. (This term comes from the Plum Village Buddhist tradition.)

aliveness

Promotes authentic self-expression, connection, and joy. Contributes to being fully present and engaged in the world. Dissolves “should”s, fear, and other emotional blockers, constrictions, and hindrances.

Some of my past and current work includes

Key
* = active
( ) = worked on but not first author

Visualization Languages

postrational

Visualization languages aim to construct mappings between data – formal representations – and visual representations, which are perceived intuitively. To what extent can such mappings be systematized?

interbeing

How can we view charts, diagrams, notations, and other visual representations as clusters in the larger space of graphic representations? What shared infrastructure and theory can we build?

How far can we take the congruence principle? What happens if we adopt the PL stance that everything is data (including functions, code, etc.)?

aliveness

How do we make visualization languages that are truly malleable and expressive? What if a visualization library optimized for whimsy and character instead of (just) precision, correctness, and efficiency?

Accessibility

  • *Mantis: A screen magnification tool
  • *Benthic: A screen reader tool for charts and diagrams
  • (*Olli): An extensible visualization library for screen reader accessibility

postrational

To what extent can a graphic representation be modeled as a perceptual data structure a la Larkin & Simon 1987?

interbeing

What can we learn from co-designing visual and accessible representations? Are there meaningful, visual-agnostic ways to think about visualization? What can we learn about the nature of spaces and encodings by examining other sensory modalities? What do Gestalt relations look like in other modalities?

aliveness

How do we allow more people to enjoy the benefits of visualization? How do we allow more people to read and write them?

What affordances do modalities besides vision have for external representations? Touch? Sound? How is information or intent conveyed through these channels?

Semi-Formal Programming

  • *Semi-Formal Notebooks: A notebook for semi-formal programming
  • Twofish: A direct manipulation editor for Bluefish

postrational

Until recently, programming languages and environments have only been able to reason about formal artifacts/representations: types, syntax, semantics, etc. With the invention of foundation models, we can more easily build general-purpose systems that can leverage intuitive methods. How do we build systems around those ideas? How do we leverage existing best-practice intuitive methods? When are they more effective than formal ones? Where do they break down?

interbeing

How does this expand our notion of what a program is? What if a program wasn’t just text but also contained visual artifacts? What if programs were ambiguous, inconsistent, or otherwise incomplete, but could still be executed? There’s been lots of exciting work in this area, before foundation models, but now we can actually execute/reason about the “intuitive” parts instead of just executing/reasoning around them.

aliveness

How might this allow us to extend the benefits of programming to more people? Code assistants like GitHub Copilot and Cursor seem to be a qualitative step change in the way we write code. Now it’s possible to write code in a much looser, freer way that’s closer to a continuous flow state. Can we extend this to more people and domains?