Interactive Video in TouchDesigner

  • TouchDesigner
  • Interactive Video
  • Audio
  • Video

This summer I took the Interactive Video in TouchDesigner course at DadaLab. These are my class notes for this six week course. Each week has a lab associated with it, with a set of instructions to apply what we're learning that week. I'm not including the lab instructions in here so as to protect the course material. These are just some rough notes on TouchDesigner foundations. Overall, this class was a great introduction to the basic concepts needed to get going in TouchDesigner and I'm very happy that I took it!

What is TouchDesigner?

TouchDesigner is a node-based programming language/visual programming language environment.

When you start TouchDesigner, you see the network editor on the right, and the palette browser on the left. Close the palette browser by clicking the x at its upper right corner to get more space.

Operators are generally synonymous with nodes in TouchDesigner. An operator outputs data to other operators. Each operator gets customized with parameters and flags.

  • To be specific, an operator is a specific entity in a network, and a node is generic, but again these are typically used interchangeably.

There are seven families of operators:

  • COMPs - Components: Object Components (3D objects), Panel Components (2D UI gadgets), and other component types. Components contain other operators.
  • TOPs - Texture Operators: used for rendering 2D images (even if source is a 3D object)
  • CHOPs - Channel Operators: used to generate numerical data.
  • POPs - Point Operators: used for 3D points, primitives, polygons, etc. Uses the GPU, so a bit more performant; replacing SOPs.
  • DATs - Data Operators: used ASCII text in various formats
  • MATs - Material Operators: used for materials and shaders
  • SOPs - Surface Operators: the legacy 3D points operator (for polygons), being replaced by the more performant POPs but still has some capabilities not in POPs

This class session will introduce us to TOPs, CHOPs, and COMPs

Some hotkeys/shortcuts/quality of life stuff:

  • The tab key will open up the window where you can select your operator
  • When you click on an operator in the network and press the p key, that will open up that operator's parameters window.
  • Value ladder: Click and hold on a numeric parameter, like radius in a Circle TOP. Some numbers will pop up (10, 1, .1, .01, and .001). Still holding your mouse click, hover over one of those values and drag your mouse right or left to increase or decrease the value by that selected amount

Week One Lab Objectives

  • Learn how to add operators to the network, connect operators to one another, and adjust their parameters.
  • Ramp TOP: operator that creates ramps (aka gradients)
    • Resolution parameter, found in Common > Resolution
    • Type parameter, found in Ramp; lets you choose a linear (vertical or horizontal), radial, or circular ramp
    • Phase parameter, found in Ramp; offsets the beginning of the ramp
      • We can give this an absolute value, 0 or 1 or something else
      • We can also animate this by giving it a constantly-increasing value using absTime.seconds
      • We can reverse the animation by giving it a constantly-decreasing value using -absTime.seconds
      • We can slow or speed up the animation using multiplication; absTime.seconds * .5 or absTime.seconds * 1.5
    • Period paremeter, found in Ramp: adjusts the length of the ramp
    • Extend Left parameter, found in Ramp: Sets the extend (or repeat) conditions of the ramp beyond the defined range...kind of answers the question, "When the ramp ends on the left edge, what happens? Is that a hard, defined line?" In this case, we choose mirror, which gives it a more blurred demarcation.
    • Hue, Saturation, Value, and Alpha parameters: adjust the color of the currently selected stop
      • Add more stops by clicking to add small boxes to the ramp range above the HSVA parameter controls
      • Change HSVA for different stops by selecting the little box you want to change
  • Circle TOP: operator that creates a circle shape
    • Radius parameter, found in Circle: adjust the size of the radius; click/hold on the + to the left of the parameter name to use the value ladder to adjust both X and Y axes together
  • Composite TOP: operator that combines multiple inputs using a composite operation
  • RGBKey TOP
  • LFO CHOP: operator that generates oscillating values, by default between -1 and 1
    • Frequency parameter: will change the frequency (speed) of the oscillation
  • Math CHOP: performs mathematics operations on channels; this is useful for changing the -1 / 1 min and max of the LFO CHOP into a different set of values (ex. 0.2 / 0.3) using the From Range and the To Range parameters
  • Null CHOP: a placeholder chop that doesn't change the data coming in; typically used at the end of a chain of channel operations to assist in exporting/activating those values for other operators to consume
    • Activate this value by clicking the + in the lower right hand of the Null node, then clicking, holding, and dragging into the node/parameter that you want this value to be passed to, pass it as a "CHOP reference"

Exporting a video

  1. Attach a Movie File Out node to the end of your network
  2. Adjust End and REnd to determine the length of the video you want to record; 600 = 10 seconds, 1200 = 20 seconds
  3. Hit the double back arrow button to scrub to the beginning of the video and stop it
  4. To the right of the play/pause/step forward + backward arrow buttons, change *Range Limit to Once
  5. At the top center of the window, unclick the Realtime button
  6. In the Movie File Out node parameters
  7. change Video Codec to MPEG 4
  8. click the + next to the File parameter and select file name and save location
  9. toggle the Record parameter to On
  10. Hit the play > button on the bottom of the window. It will play for 20 seconds and that recording will get saved where you specified.

Quiz

Question 1: How can you access the Operator families? Answer 1: The tab key

Question 2: What letter on the keyboard can I use to access a Node's parameters? Answer 2: The P key

Question 3: What is a pythom command to give a parameter perpetual animation? Answer 3: absTime.seconds or absTime.frame

Question 4: True or false? "absTime.seconds" is case-sensitive. Answer 4: True

Question 5: True or false? You can plug a CHOP into a TOP. Answer 5: False

Week Two

Today's class: Building a Rendering Network and Feedback Loops

  • New Operator Family: POPS
  • POPS use the GPU, so it is a bit more performant
  • SOPS use the CPU
  • For POPS, use absTime.frame instead of absTime.seconds

Feedback Concepts / Techniques

Feedback paints a picture using Time and Recursion. Feedback uses a system's output as its new input, creating a recursive, evolving visual loop.

At its essence, feedback is using a system's output as its new input, creating a recursive, evolving visual loop.

Time and recursion paint the picture. The feedback TOP is an operator for generating recursive, self-referential visual effects. This TOP captures the current frame, stores int, and then feeds it back into the system to be processed and mixed with the incoming frames. It continuously blends past and present frames.

Some feedback types:

  • Motion trails
  • Infinite mirror
  • Reaction diffusion

Rendering

Rendering is turning 3D objects into 2D textures. There is a common technique for this called the Render Network.

How to build a Render Network:

  1. Add a 3D node, like a Box POP
  2. Send that to a Geometry COMP
  3. Add a Camera COMP, a Light COMP, and a Render TOP

Add some motion to your Box.

How to build a Feedback Loop. This follows your Render TOP:

  1. Add a Feedback TOP. Connect the Render TOP to the Feedback TOP
  2. Add a Level TOP
  3. Connect the Feedback TOP to the Level TOP
  4. Add a Composite TOP
  5. Connect the Render TOP to the Composite TOP's first input
  6. Connect the Level TOP to the Composite TOP's second input
  7. Change the Composite TOP's Operation to "over"
  8. Pass the name of Composite TOP (something like comp1 by default) to the Feedback TOP's Target TOP parameter
  9. Adjust the Level TOP's Opacity parameter down

Quiz

Question 1: If you want to make a feedback loop, which operator should you start with first? Answer 1: Feedback TOP

Question 2: Which TOP allows you to adjust the Opacity of the feedback? Answer 2: Level TOP

Question 3: I'm going to render a Box POP. First, I plug the Box POP into a Geometry COMP. What other components / nodes do I need to complete the rendering network? Answer 3: Camera COMP, Light COMP, and Render TOP

Week Three

Challenge 1

  1. Animate a Noise TOP on the Z-axis
  2. Give it a resolution of 1280 x 720
  3. Display it in the network's background

Challenge 2

  1. Create a Render Network
  2. Start with a "Box" SOP or POP
  3. Reduce the box size (scale) to 0.3
  4. Make the box move from Left to Right
  5. Make the box rotate on the Y-axis

Today's concepts

  1. Points
  2. Attributes
  3. Instancing

New nodes for today:

  1. Copy POP
  2. Attribute POP

What are points?

Points are the fundamental building blocks of 3D geometry within a SOP (Surface Operator) or POP (Point Operator) context.

  • A point is a single 3D location in space (X, Y, Z)
  • By itself, a point has no size, surface, or volume; it's just a coordinate
  • When you have many points, they can be connected to form primitives (polygons, lines, or meshes);

Each point can store:

  • Position (P): the XYZ coordinates
  • Color:* RGB values for shading/display
  • Normal (N): Direction for lighting

How are points used?

  • Creating shapes: A grid of points becomes a terrain; a line of points becomes a curve
  • Particle systems: Each particle is a point moving over time
  • Instancing: A point can "instance" (copy) a larger piece of geometry, using the point's position, color, or other attributes to control the instance.
  • Data processing: You can move, filter, or manipulate points with SOPs like Point, Transform, Noise, or Attribute Create

What are attributes?

Attributes are pieces of data attached to the points or promitives (polygons, lines, meshes) of a POP or SOP. They let you store extra information beyond just position.

Common attributes:

  • Position (P): this is a point's position in 3D space; X, Y, Z
  • Color (Color): surface color and alpha specified by RGBA. Values range from 0 to 1
  • Normal (N): a normal is a directional vector associated with a particular geometric entity. The normal to a surface at a given point is a vector perpendicular to the surface at that point

What is a "vector" in this context?

  • A point describes a location, but a Vector, such as a Normal, describes movement, force, or direction
  • A good way to think of it is: Which way will light reflect off of a surface?

What is instancing?

Instancing is a technique where you take a single piece of geometry (a "template") and display it many times, with each copy (an "instance" controlled by a point and its attribues).

  • Template: the thing you copy
  • Attributes: how to customize each copy or instance (such as color, scale, rotation)

Week Four

No slides this week, but we focused on audio reactivity. We started with an Audio File In CHOP, which has a default audio file it loads that we can use. We can connect this to an Audio Device Out CHOP to actually hear the song.

We can also connect the Audio File In CHOP to audioAnalysis (in Palette > Tools > audioAnalysis). This lets us fine-tune things like high, mid, and low sounds, and output those values as they shift based on the audio. A combination of select and math helps us normalize those values and pass them as references to other operator parameters, where they can control our visuals, like the size of an instanced POP or the z-translate of a Noise TOP.

Week Five

  • Common instancing error: "All OPs must have the same number of instances".
  • If we are working with POPs, CHOPs, and TOPs, how can we make sure all OPs have the same number of instances?
    • POPs: Instances are generated with Points
    • CHOPs: Instances are generated with Samples (usually a Length parameter)
    • TOPs: Instances are generated with Pixels (Resolution - usually x and y axes)

With that info hammered, out, this week's project focused on instancing using TOPs.

Week Six

This week's lesson focused on using the MediaPipe plugin to control parameters. This is a cool plugin that tracks things like hand or facial movement. Similar to audio analysis, we can use the Select CHOP to focus in on specific gestures, like a pinching motion, an eyebrow raise, or hand position and pass those value changes to other operator parameters in our network.

Inspiration