Because my abstract was not super detailed, here’s the intro for your perusal:
Body Paint will interactively create a virtual painting based on the physical properties (path, velocity, acceleration, and angular velocity) of the movements of the dancer. The default process will do a “conventional” painting – movements will map to a “canvas” at the front of the space. Ideally, there will be two other mappings: a painted “forest” where the motions will be traced and marked in place, with strokes applied, and a moveable cross-section will be projected; and a footwork painting, which only paints from the motion of the footsteps on the ground. These strokes will be selected at the beginning of the piece from a small group: oil paintbrush, ink/calligraphy, and charcoal/pastel. These may also be changed in the resulting painting after the fact. The color of the medium will be interpreted based solely on these properties, although the spectrum can be limited t the dancer’s specifications.
Now, some elaboration and new knowledge:
To begin with, a little while ago, I met with Joe and Alla to go over my design doc and my basic idea. Some things I did not know/have reconsidered/adjusted:
Motion Data
CMU has an exquisite database of captured motions, including, dance motions! (http://mocap.cs.cmu.edu/ – note subject #5) This will be a great resource when testing out strokes and mappings. It has also been helpful (and will definitely continue being helpful) for figuring out the Vicon system and its methods.
Data Format
Before I can really get to mapping the movement to a path and applying a stroke, I need to think about file types. I’ve never worked with recorded mocap data, just real-time, so I didn’t think too deeply about how the data was stored. I knew the system located a series of hyper-reflective balls in 3D space. I now know this data is then translated into joint angles for most later levels of processing. I am currently leaning to the raw-er data of c3d, with simply the positions of the markers. However, I need to figure out how to read the data in any file type used. CMU has an .asf/.amc reader. The software “ViconIQ” that comes with the system processes the camera data and ultimately outputs a .vsk/.v. I am unsure as of now in what data formats Vicon captures/processes/records in real time and which one is ideal.
User Interface
First and foremost, I should begin planning this out presently, not toward the end was originally prescribed. To that end:
I do not want it to use gesture recognition. Gesture recognition requires new learning that is orthogonal to the actual goal (creating a virtual painting by dancing). Instead, I would like a conventional and usable interface on the home machine of the software, something simple and intuitive for the accepted, learned interaction with GUIs. It will be a short series of choice:
- What stroke would you like to use?: Oil paint, calligraphic ink, charcoal or pastels
- What color range? Select range of hue, saturation, and value
- What kind of mapping would you like to use? Conventional canvas(movements are interpretted as strokes on a virtual canvas at the “front” of the 3D dance- space, regardless of “depth”), 3D painting (paths follow the movement in 3D space with a moveable viewing plane that can cut through the space anywhere), or footwork (only motion that contacts the floor will leave a path, resulting in something more like pointilism)
These will be a conventional, simple interface on the base machine. Once the dancer has selected her choices, she will get in place to begin her piece, and the technician will select an option to start recording.
After completing the piece, I would like the dancer to be able to change the stroke or color selection to see other resulting paintings.
Notes:
-Joe recommended using prerecorded bvh files (such as those available from CMU) to show off the various strokes in the interface.
-If I get different mappings working as intended, I want to demo the resulting “painting” in the interface – ie, a 2D painting, a 3D painting, or a “floor” painting from the same dance.
User Study
-In my head, I assumed I would ask people to come in and try it out and give feedback. I should clearly formalize this and include it in my deliverables. This is a project about user experience, after all.
QUESTIONS:
Brushstroke Algorithms:
-I am having some trouble finding useful papers for simple implementation of stroke algorithms. I find all these papers on painterly rendering which seems more complex than required, and some on different interfaces or applications, but none simply on taking a path and applying an artistic stroke, a la Photoshop and Illustrator. I remember some things on defining curves from Intro Graphics which I could probably use to smooth the strokes, but regardless, I need algorithms for the strokes so I can get onto implementing them. I’d appreciate any help parsing through the vast literature on this and related topics to narrow down to very relevant topics. I will keep looking, but I feel like these are pretty standard, I am just not exposed to them.
I am a bit behind, but I have reworked my schedule to optimize catching up and getting back on track.
