I modified JC Testud's Video Generation With pix2pix to do things like predict subsequent frames in realtime and gradually fade in separate feeds. For Aleph Earth I created a Unity app that would receive audio analysis by Zachary Boyt over OSC which was visualized and fed into Pix2PixHD. Through this process we used the choirs recording to guide abstracted visions of ecological catastrophy.
Video of the realtime setup in action.
There's some nice writing about its inception on the University of Oregon's AroundTheO blog.