5. Interface & Sound


Content

1. Sound in Android Mode
2. Interaction, Controller, HyperInstruments
3. Game Controller as instruments
4. Leap Motion Detection as interface
5. Using Leap Motion to control Android Devices
6. Processing with MAX
7. Microsoft Kinect as an interface




1. Sound in Android Mode

Processing`s sound library isn`t yet ported for Android Mode, so sound output isn`t achievable out of the box. However here is an explenation for those who want to go for it anyway: Here.

Bloom, Brian Eno, 2008




2. Interaction, Controller, HyperInstruments

Example: Karlheinz Essl, Pachinko for toy piano and computer, 2013, http://www.essl.at/works/pachinko.html




3. Game Controller as instruments (e.g. XBox One Controller)

Download the Driver for OSX here (don`t get irritated – the "360"-driver will work).



Library for Processing: Game Control Plus. Or load it from the Contribution Manager of Processing.
Video Tutorial on how to configure a contoller for developers is here.

Create a configuration file for your controller

As a first step you will want to create a configuration file for your controller. We use the sketch provided by the library in Documents/Processing/libraries/GameControlPlus/examples/Configurator.
After you created the mapping you have throw the config-file into the data-folders of your upcoming sketches.




Examples

import net.java.games.input.*;
import org.gamecontrolplus.*;
import org.gamecontrolplus.gui.*;

ControlIO control;
ControlDevice stick;

float lx, ly, rx, ry, b1, b2;

void setup(){
   size(800,600);
   colorMode(HSB);
   background(0);
   fill(0,20);
   stroke(0,0,0,10);
   
   control = ControlIO.getInstance(this);
   stick = control.getMatchedDevice("XBOXbasic");
   
   if (stick == null){
    println("nothing found");
    System.exit(-1); 
   }
}

void getControllerInput(){
    lx = map(stick.getSlider("STICK1X").getValue(),-1,1,0, width);
    ly = map(stick.getSlider("STICK1Y").getValue(),-1,1,0, height);
    rx = map(stick.getSlider("STICK2X").getValue(),-1,1,40, 80);
    ry = map(stick.getSlider("STICK2Y").getValue(),-1,1,0, 255);
    b1 = map(stick.getSlider("BUTTON1").getValue(),-1,1,-1, 1);
    b2 = map(stick.getSlider("BUTTON2").getValue(),-1,1,-1, 1);
    println(rx);
}

void draw(){
    getControllerInput();
    stroke(rx, 100, 255, 10);
    line(0,0, lx, ly);
    line(width, 0, lx, ly);
    line(0,height, lx, ly);
    line(width,height, lx, ly);
}
        
        

Exercise (20 minutes):
Come up with your own visualization. Use data from both sticks and from a button.





Installation at Intrepid Museum, New York


4. Leap Motion Detection as interface

Leap Motion in Processing

Library Leap Motion for Processing installieren.

The documentation of the API provided by Leap Motion is accessible in the developer sectionof their website.

A tutorial explaining the use of Leap Motion can be found here.

Leap Motion offers a huge amount of RAW data that can be used for further individual processing.

Example: Using Leap Motion


  import de.voidplus.leapmotion.*;

LeapMotion leap;

void setup() {
  size(800, 500, OPENGL);
  leap = new LeapMotion(this);
}

void draw() {
  background(255);
  int fps = leap.getFrameRate();
  text(fps + " fps", 20, 20);

  for (Hand hand : leap.getHands ()) {
    hand.draw();
  }
}
  

Example: Hand-Painting with Leap Motion

import de.voidplus.leapmotion.*;

LeapMotion leap;

float INTERACTION_SPACE_WIDTH = 200; // left-right from user
float INTERACTION_SPACE_DEPTH = 150; // away-and-toward user
float colorShift = 0;
float thbDistance, idxDistance, mdlDistance, rngDistance, pkyDistance;

void setup() {
  size(690, 500, P3D);
  leap = new LeapMotion(this);
  colorMode(HSB, 255);
  background(0);
}

void draw() {
  noStroke();
  fill(colorShift,255,30, 2);
  rect(0,0,width,height);
  if (colorShift < 215){
    colorShift += 1.5;
  }else{
    colorShift = 0;
  }
  int fps = leap.getFrameRate();

  for (Hand hand : leap.getHands ()) {
    PVector handPos = hand.getRawPosition();
    PVector thumbTip = hand.getThumb().getRawPositionOfJointTip();
    PVector indexTip = hand.getIndexFinger().getRawPositionOfJointTip();
    PVector middleTip = hand.getMiddleFinger().getRawPositionOfJointTip();
    PVector ringTip = hand.getRingFinger().getRawPositionOfJointTip();
    PVector pinkyTip = hand.getPinkyFinger().getRawPositionOfJointTip();
    
    thbDistance = PVector.dist(handPos, thumbTip);          // Calculate the distance of fingertips to palm
    idxDistance = PVector.dist(handPos, indexTip);
    mdlDistance = PVector.dist(handPos, middleTip);
    rngDistance = PVector.dist(handPos, ringTip);
    pkyDistance = PVector.dist(handPos, pinkyTip);
    println(thbDistance);
    handleFinger(thumbTip, "thb");
    handleFinger(indexTip, "idx");
    handleFinger(middleTip, "mdl");
    handleFinger(ringTip, "rng");
    handleFinger(pinkyTip, "pky");
  }
}

void handleFinger(PVector pos, String id) {
  float radius = 0;
  // map finger tip position to 2D surface
  float x = map(pos.x, -INTERACTION_SPACE_WIDTH,
  INTERACTION_SPACE_WIDTH, 0, width);
  float y = map(pos.z, -INTERACTION_SPACE_DEPTH,
  INTERACTION_SPACE_DEPTH, 0, height);
  colorMode(HSB, 255);
  if (id == "thb"){
    fill(0+colorShift,100,255,20);
    radius = pow(thbDistance,3)/10000;
  }else if (id == "idx"){
    fill(10+colorShift,100,255,20);
    radius = pow(idxDistance,3)/10000;
  }else if (id == "mdl"){
    fill(20+colorShift,100,255,20);
    radius = pow(mdlDistance,3)/10000;
  }else if (id == "rng"){
    fill(30+colorShift,100,255,20);
    radius = pow(rngDistance,3)/10000;
  }else if (id == "pky"){
    fill(40+colorShift,100,255,20);
    radius = pow(pkyDistance,3)/10000;
  }
  
  noStroke();
  ellipse(x, y, radius, radius);

}
  

Leap Motion is also capable of detection some gestures. See the example in the library`s folder!




5. Using Leap Motion to control Android Devices

Gesture to OSC

We`ll again use Open Sound Control (as seen above) to communicate with Android tablets. On the tablets we have sketches running that are listening for our OSC messages.


  	

In the App for the Devices we define how the incoming data should be translated to audiovisual output.


  	



6. Processing with MAX

Mew, Emily Groves

Creating instruments in MAX

The rather complex synthesizer ›X.FM‹ comes with MAX7

Exercise (20 minutes):
Build your own tiny MAX-instrument. Use the objects saw~, etc.

Exchange data with Open Sound Control (OSC)

"Open Sound Control (OSC) is a protocol for communication among computers, sound synthesizers, and other multimedia devices that is optimized for modern networking technology. Bringing the benefits of modern networking technology to the world of electronic musical instruments, OSC's advantages include interoperability, accuracy, flexibility, and enhanced organization and documentation." (opensoundcontrol.org)

Exchange Data between Processing and MAX

MAX patch:

import netP5.*;
import oscP5.*;

OscP5 osc;
NetAddress myRemoteLocation; 

void setup(){
  size(800, 600);
  osc = new OscP5(this, 12000);
  myRemoteLocation = new NetAddress("127.0.0.1", 13000);
}

void draw(){
  
  
}


void mousePressed(){
 OscMessage myMessage = new OscMessage("hello");
 myMessage.add("world!");
 osc.send(myMessage, myRemoteLocation);
 println(myMessage);
}
  

// Libs for contraoller
import net.java.games.input.*;
import org.gamecontrolplus.*;
import org.gamecontrolplus.gui.*;

// Libs for OSC communication with MAX
import netP5.*;
import oscP5.*;

ControlIO control;
ControlDevice stick;
OscP5 osc;
NetAddress myRemoteLocation; 

float lx, ly, rx, ry, b1, b2;

float x,y;
boolean b = false;

PVector Walker;
PVector SourceOne, SourceTwo, SourceThree;
float distOne, distTwo, distThree;


void setup() {
  size(1280, 720, P3D);
  colorMode(HSB);
  
  control = ControlIO.getInstance(this);
  stick = control.getMatchedDevice("XBOXbasic");
  osc = new OscP5(this, 12000);
  myRemoteLocation = new NetAddress("127.0.0.1", 13000);
  SourceOne = new PVector(40, 380);
  SourceTwo = new PVector(600, 200);
  SourceThree = new PVector(900, 550);
  Walker = new PVector(width/2, height/2);
}

void draw() {
  // 1. get controller data input
  getControllerInput();
  
  // 2. do the (simple) algorithms
  Walker.set(lx, ly);                // control walker with left stick
  distOne = Walker.dist(SourceOne);
  distTwo = Walker.dist(SourceTwo);
  distThree = Walker.dist(SourceThree);
  
  
  // 4. map the data to the parameters of ellipses and do the visuals
  background(0);
  ellipse(Walker.x,Walker.y,10,10);
  stroke(0, 0,255, 100);
  fill(20,255,255,50);
  ellipse(SourceOne.x, SourceOne.y, 2*distOne,2*distOne);
  fill(35,255,255,50);
  ellipse(SourceTwo.x, SourceTwo.y, 2*distTwo, 2*distTwo);
  fill(50,255,255,50);
  ellipse(SourceThree.x, SourceThree.y, 2*distThree, 2*distThree);
  stroke(0);
  fill(0,0,255);
  ellipse(Walker.x,Walker.y,10,10);
  
  // send OSC to MAX
  OscMessage myMessage = new OscMessage("");
  myMessage.add(Walker.x+" "+Walker.y+" "+distOne+" "+distTwo+" "+distThree);
  osc.send(myMessage, myRemoteLocation);

}

void getControllerInput(){
    lx = map(stick.getSlider("STICK1X").getValue(),-1,1,0, width);
    ly = map(stick.getSlider("STICK1Y").getValue(),-1,1,0, height);
    rx = map(stick.getSlider("STICK2X").getValue(),-1,1,40, 80);
    ry = map(stick.getSlider("STICK2Y").getValue(),-1,1,0, 255);
    b1 = map(stick.getSlider("BUTTON1").getValue(),-1,1,-1, 1);
    b2 = map(stick.getSlider("BUTTON2").getValue(),-1,1,-1, 1);
}



  
{
	"patcher" : 	{
		"fileversion" : 1,
		"appversion" : 		{
			"major" : 7,
			"minor" : 3,
			"revision" : 3,
			"architecture" : "x86",
			"modernui" : 1
		}
,
		"rect" : [ 303.0, 225.0, 993.0, 893.0 ],
		"bglocked" : 0,
		"openinpresentation" : 0,
		"default_fontsize" : 12.0,
		"default_fontface" : 0,
		"default_fontname" : "Arial",
		"gridonopen" : 1,
		"gridsize" : [ 15.0, 15.0 ],
		"gridsnaponopen" : 1,
		"objectsnaponopen" : 1,
		"statusbarvisible" : 2,
		"toolbarvisible" : 1,
		"lefttoolbarpinned" : 0,
		"toptoolbarpinned" : 0,
		"righttoolbarpinned" : 0,
		"bottomtoolbarpinned" : 0,
		"toolbars_unpinned_last_save" : 0,
		"tallnewobj" : 0,
		"boxanimatetime" : 200,
		"enablehscroll" : 1,
		"enablevscroll" : 1,
		"devicewidth" : 0.0,
		"description" : "",
		"digest" : "",
		"tags" : "",
		"style" : "",
		"subpatcher_template" : "",
		"boxes" : [ 			{
				"box" : 				{
					"format" : 6,
					"id" : "obj-69",
					"maxclass" : "flonum",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 825.0, 478.0, 50.0, 22.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"format" : 6,
					"id" : "obj-68",
					"maxclass" : "flonum",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 496.5, 446.5, 50.0, 22.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"fontname" : "Arial",
					"fontsize" : 13.0,
					"id" : "obj-64",
					"maxclass" : "message",
					"numinlets" : 2,
					"numoutlets" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 61.0, 665.0, 43.0, 23.0 ],
					"style" : "",
					"text" : "clear"
				}

			}
, 			{
				"box" : 				{
					"attr" : "edit_mode",
					"id" : "obj-63",
					"maxclass" : "attrui",
					"numinlets" : 1,
					"numoutlets" : 1,
					"orientation" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 610.0, 459.0, 75.0, 44.0 ],
					"presentation_rect" : [ 0.0, 0.0, 0.0, 44.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-62",
					"maxclass" : "newobj",
					"numinlets" : 6,
					"numoutlets" : 1,
					"outlettype" : [ "signal" ],
					"patching_rect" : [ 177.5, 707.0, 71.5, 22.0 ],
					"style" : "",
					"text" : "biquad~"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-61",
					"maxclass" : "newobj",
					"numinlets" : 2,
					"numoutlets" : 1,
					"outlettype" : [ "int" ],
					"patching_rect" : [ 511.5, 318.0, 56.5, 22.0 ],
					"style" : "",
					"text" : "* 5"
				}

			}
, 			{
				"box" : 				{
					"fontface" : 0,
					"id" : "obj-60",
					"maxclass" : "filtergraph~",
					"nfilters" : 1,
					"numinlets" : 8,
					"numoutlets" : 7,
					"outlettype" : [ "list", "float", "float", "float", "float", "list", "int" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 590.0, 513.0, 254.0, 136.0 ],
					"setfilter" : [ 0, 3, 1, 0, 0, 1485.0, 3.540734, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-58",
					"maxclass" : "newobj",
					"numinlets" : 2,
					"numoutlets" : 1,
					"outlettype" : [ "int" ],
					"patching_rect" : [ 371.5, 361.0, 29.5, 22.0 ],
					"style" : "",
					"text" : "/ 5"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-57",
					"maxclass" : "newobj",
					"numinlets" : 2,
					"numoutlets" : 1,
					"outlettype" : [ "int" ],
					"patching_rect" : [ 441.5, 361.0, 53.0, 22.0 ],
					"style" : "",
					"text" : "/ 10"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-55",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 811.5, 235.0, 161.0, 20.0 ],
					"presentation_rect" : [ 811.5, 235.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "data from processing"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-54",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 795.5, 813.0, 161.0, 20.0 ],
					"presentation_rect" : [ 795.5, 813.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "output"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-53",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 811.5, 421.0, 161.0, 20.0 ],
					"presentation_rect" : [ 811.5, 420.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "mapping"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-52",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 811.5, 318.0, 161.0, 20.0 ],
					"style" : "",
					"text" : "calculations"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-50",
					"maxclass" : "newobj",
					"numinlets" : 2,
					"numoutlets" : 1,
					"outlettype" : [ "int" ],
					"patching_rect" : [ 441.5, 318.0, 53.0, 22.0 ],
					"style" : "",
					"text" : "- 400"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-49",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 511.5, 274.0, 79.0, 20.0 ],
					"presentation_rect" : [ 683.5, 274.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "distThree"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-48",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 441.5, 274.0, 53.0, 20.0 ],
					"presentation_rect" : [ 612.5, 274.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "distTwo"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-47",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 371.5, 274.0, 53.0, 20.0 ],
					"presentation_rect" : [ 541.0, 274.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "distOne"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-46",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 288.0, 274.0, 53.0, 20.0 ],
					"presentation_rect" : [ 457.0, 274.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "walkerY"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-44",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 196.0, 274.0, 53.0, 20.0 ],
					"style" : "",
					"text" : "walkerX"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-43",
					"linecount" : 2,
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 38.0, 212.0, 112.0, 33.0 ],
					"style" : "",
					"text" : "print received data to console"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-42",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 548.0, 437.0, 125.5, 20.0 ],
					"presentation_rect" : [ 689.0, 350.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "minimum loop point"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-41",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 427.5, 437.0, 112.0, 20.0 ],
					"presentation_rect" : [ 571.5, 350.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "playback speed"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-40",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 301.0, 437.0, 112.0, 20.0 ],
					"presentation_rect" : [ 446.0, 350.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "Start at X ms"
				}

			}
, 			{
				"box" : 				{
					"format" : 6,
					"id" : "obj-39",
					"maxclass" : "flonum",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 548.0, 464.0, 50.0, 22.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"format" : 6,
					"id" : "obj-38",
					"maxclass" : "flonum",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 427.5, 464.0, 50.0, 22.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"format" : 6,
					"id" : "obj-34",
					"maxclass" : "flonum",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 301.0, 464.0, 50.0, 22.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-33",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 1,
					"outlettype" : [ "signal" ],
					"patching_rect" : [ 427.5, 513.0, 33.0, 22.0 ],
					"style" : "",
					"text" : "sig~"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-32",
					"maxclass" : "message",
					"numinlets" : 2,
					"numoutlets" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 168.5, 464.0, 43.0, 22.0 ],
					"style" : "",
					"text" : "loop 1"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-31",
					"maxclass" : "gain~",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "signal", "int" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 168.5, 753.0, 119.0, 40.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-27",
					"maxclass" : "newobj",
					"numinlets" : 3,
					"numoutlets" : 2,
					"outlettype" : [ "signal", "signal" ],
					"patching_rect" : [ 177.5, 665.0, 119.0, 22.0 ],
					"saved_object_attributes" : 					{
						"basictuning" : 440,
						"followglobaltempo" : 0,
						"formantcorrection" : 0,
						"loopend" : [ 0.0, "ms" ],
						"loopstart" : [ 681.0, "ms" ],
						"mode" : "basic",
						"originallength" : [ 0.0, "ticks" ],
						"originaltempo" : 120.0,
						"phase" : [ 0.0, "ticks" ],
						"pitchcorrection" : 0,
						"quality" : "basic",
						"timestretch" : [ 0 ]
					}
,
					"style" : "",
					"text" : "groove~ loopsample"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-26",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 610.0, 161.0, 150.0, 20.0 ],
					"presentation_rect" : [ 32.0, 354.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "load sample",
					"textjustification" : 0
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-25",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 168.5, 437.0, 112.0, 20.0 ],
					"style" : "",
					"text" : "turn looping on",
					"textjustification" : 0
				}

			}
, 			{
				"box" : 				{
					"fontname" : "Arial",
					"fontsize" : 13.0,
					"id" : "obj-16",
					"maxclass" : "number",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 511.5, 235.0, 53.0, 23.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"fontname" : "Arial",
					"fontsize" : 13.0,
					"id" : "obj-15",
					"maxclass" : "number",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 441.5, 235.0, 53.0, 23.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"fontname" : "Arial",
					"fontsize" : 13.0,
					"id" : "obj-14",
					"maxclass" : "number",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 371.5, 235.0, 53.0, 23.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-13",
					"maxclass" : "message",
					"numinlets" : 2,
					"numoutlets" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 610.0, 69.0, 162.0, 22.0 ],
					"style" : "",
					"text" : "read"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-56",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 321.0, 100.0, 116.0, 22.0 ],
					"style" : "",
					"text" : "tosymbol"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-51",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 321.0, 133.0, 116.0, 22.0 ],
					"style" : "",
					"text" : "fromsymbol"
				}

			}
, 			{
				"box" : 				{
					"fontname" : "Arial",
					"fontsize" : 13.0,
					"id" : "obj-45",
					"maxclass" : "number",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 288.0, 235.0, 53.0, 23.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"fontname" : "Arial",
					"fontsize" : 13.0,
					"id" : "obj-3",
					"maxclass" : "number",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 196.0, 235.0, 53.0, 23.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-19",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 5,
					"outlettype" : [ "float", "float", "float", "float", "float" ],
					"patching_rect" : [ 321.0, 167.0, 116.0, 22.0 ],
					"style" : "",
					"text" : "unpack 0. 0. 0. 0. 0."
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-9",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 42.0, 167.0, 34.0, 22.0 ],
					"style" : "",
					"text" : "print"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-12",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 321.0, 66.0, 116.0, 22.0 ],
					"style" : "",
					"text" : "udpreceive 13000"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-4",
					"maxclass" : "ezdac~",
					"numinlets" : 2,
					"numoutlets" : 0,
					"patching_rect" : [ 168.5, 824.0, 45.0, 45.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-1",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "float", "bang" ],
					"patching_rect" : [ 610.0, 123.0, 162.0, 22.0 ],
					"style" : "",
					"text" : "buffer~ loopsample anton.aif"
				}

			}
 ],
		"lines" : [ 			{
				"patchline" : 				{
					"destination" : [ "obj-56", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"order" : 0,
					"source" : [ "obj-12", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-9", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 330.5, 90.0, 51.5, 90.0 ],
					"order" : 1,
					"source" : [ "obj-12", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-1", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-13", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-58", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-14", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-50", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-15", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-61", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-16", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-14", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-19", 2 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-15", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-19", 3 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-16", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-19", 4 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-3", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-19", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-45", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-19", 1 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-62", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-27", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-4", 1 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 278.0, 810.0, 204.0, 810.0 ],
					"source" : [ "obj-31", 1 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-4", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-31", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-27", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 178.0, 651.0, 187.0, 651.0 ],
					"source" : [ "obj-32", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-27", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 437.0, 651.0, 187.0, 651.0 ],
					"source" : [ "obj-33", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-27", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 310.5, 651.0, 187.0, 651.0 ],
					"source" : [ "obj-34", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-33", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-38", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-27", 1 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 557.5, 651.0, 237.0, 651.0 ],
					"source" : [ "obj-39", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-57", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-50", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-19", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-51", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-51", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-56", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-38", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-57", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-31", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-58", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-62", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-60", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-60", 5 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 521.0, 423.0, 767.357117, 423.0 ],
					"source" : [ "obj-61", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-31", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-62", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-60", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-63", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-62", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-64", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-60", 7 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-69", 0 ]
				}

			}
 ],
		"parameters" : 		{

		}
,
		"dependency_cache" : [  ],
		"autosave" : 0
	}

}
  

import de.voidplus.leapmotion.*;
import netP5.*;
import oscP5.*;

OscP5 osc;
NetAddress myRemoteLocation; 
LeapMotion leap;
ArrayList haende = new ArrayList();
ArrayList indexPoints = new ArrayList();
float distance;

void setup() {
  size(690, 500, P3D);
  osc = new OscP5(this, 12000);
  myRemoteLocation = new NetAddress("127.0.0.1", 13000);
  leap = new LeapMotion(this);
  colorMode(HSB, 255);
  noStroke();
  fill(50,255,255);
  lights();
  indexPoints.add(new PVector(0,0,0));  // add Vector for left index finger
  indexPoints.add(new PVector(0,0,0));  // add Vector for right index finger
}

void draw() {
 background(255);
 
  // get all detected hands an put them in an ArrayList
 ArrayList hands = leap.getHands();
 
 // loop thorugh all detected hand-objects
 for (int i = 0; i < hands.size() ; i++){
   Hand currentHand = hands.get(i);
   PVector indexTip = currentHand.getIndexFinger().getRawPositionOfJointTip(); 
   PVector currentTip = indexPoints.get(i);
   currentTip.set(indexTip);
   
   // draw spheres at positions of index fingers
   noStroke();
   pushMatrix();
   translate(width/2, 0, 0);
   translate(indexTip.x, height-indexTip.y, indexTip.z);
   sphere(currentTip.y/10);
   sphere(20);
   popMatrix();
   
   // Send Data to MAX
   OscMessage myMessage = new OscMessage("");
   myMessage.add(distance+" "+currentTip.y);
   osc.send(myMessage, myRemoteLocation);
 }
 
  // Calculate distance between index fingers and draw the line
 if (indexPoints.size() == 2){
    println(indexPoints.size());
    PVector Pleft = indexPoints.get(0);
    PVector Pright = indexPoints.get(1);
    distance = Pleft.dist(Pright);
    stroke(0,0,0);
    strokeWeight((1000/distance));
    translate(width/2, 0, 0);
    line(Pleft.x, height-Pleft.y, Pleft.z, Pright.x, height-Pright.y, Pright.z); 
 }
}
  

 {
	"patcher" : 	{
		"fileversion" : 1,
		"appversion" : 		{
			"major" : 7,
			"minor" : 3,
			"revision" : 3,
			"architecture" : "x86",
			"modernui" : 1
		}
,
		"rect" : [ 451.0, 344.0, 918.0, 734.0 ],
		"bglocked" : 0,
		"openinpresentation" : 0,
		"default_fontsize" : 12.0,
		"default_fontface" : 0,
		"default_fontname" : "Arial",
		"gridonopen" : 1,
		"gridsize" : [ 15.0, 15.0 ],
		"gridsnaponopen" : 1,
		"objectsnaponopen" : 1,
		"statusbarvisible" : 2,
		"toolbarvisible" : 1,
		"lefttoolbarpinned" : 0,
		"toptoolbarpinned" : 0,
		"righttoolbarpinned" : 0,
		"bottomtoolbarpinned" : 0,
		"toolbars_unpinned_last_save" : 0,
		"tallnewobj" : 0,
		"boxanimatetime" : 200,
		"enablehscroll" : 1,
		"enablevscroll" : 1,
		"devicewidth" : 0.0,
		"description" : "",
		"digest" : "",
		"tags" : "",
		"style" : "",
		"subpatcher_template" : "",
		"boxes" : [ 			{
				"box" : 				{
					"format" : 6,
					"id" : "obj-30",
					"maxclass" : "flonum",
					"minimum" : 0.0,
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 274.916656, 385.0, 50.0, 22.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"format" : 6,
					"id" : "obj-29",
					"maxclass" : "flonum",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 179.833328, 383.0, 35.0, 22.0 ],
					"presentation_rect" : [ 206.5, 385.0, 0.0, 0.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-28",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 179.833328, 360.0, 53.0, 20.0 ],
					"presentation_rect" : [ 209.5, 360.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "gain"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-27",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 274.916656, 360.0, 69.0, 20.0 ],
					"presentation_rect" : [ 264.5, 354.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "centerfrq"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-26",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 357.5, 360.0, 53.0, 20.0 ],
					"presentation_rect" : [ 322.0, 354.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "Q"
				}

			}
, 			{
				"box" : 				{
					"format" : 6,
					"id" : "obj-25",
					"maxclass" : "flonum",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 357.5, 385.0, 50.0, 22.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-22",
					"maxclass" : "newobj",
					"numinlets" : 4,
					"numoutlets" : 1,
					"outlettype" : [ "signal" ],
					"patching_rect" : [ 135.0, 446.0, 153.5, 22.0 ],
					"style" : "",
					"text" : "reson~"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-10",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 414.5, 249.0, 53.0, 20.0 ],
					"presentation_rect" : [ 275.0, 232.0, 0.0, 0.0 ],
					"style" : "",
					"text" : "rightY"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-6",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 277.0, 249.0, 53.0, 20.0 ],
					"style" : "",
					"text" : "leftY"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-44",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 135.0, 247.0, 53.0, 20.0 ],
					"style" : "",
					"text" : "distance"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-7",
					"maxclass" : "gain~",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "signal", "int" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 135.0, 519.0, 111.0, 37.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-5",
					"maxclass" : "ezdac~",
					"numinlets" : 2,
					"numoutlets" : 0,
					"patching_rect" : [ 139.0, 643.0, 45.0, 45.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-2",
					"maxclass" : "newobj",
					"numinlets" : 2,
					"numoutlets" : 1,
					"outlettype" : [ "signal" ],
					"patching_rect" : [ 135.0, 383.0, 39.0, 22.0 ],
					"style" : "",
					"text" : "saw~"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-56",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 231.5, 63.0, 111.0, 22.0 ],
					"style" : "",
					"text" : "tosymbol"
				}

			}
, 			{
				"box" : 				{
					"fontname" : "Arial",
					"fontsize" : 13.0,
					"id" : "obj-53",
					"maxclass" : "number",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 414.5, 211.0, 53.0, 23.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-52",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 603.0, 122.0, 249.0, 20.0 ],
					"style" : "",
					"text" : "covert to numbers"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-51",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 231.5, 93.0, 111.0, 22.0 ],
					"style" : "",
					"text" : "fromsymbol"
				}

			}
, 			{
				"box" : 				{
					"fontname" : "Arial",
					"fontsize" : 13.0,
					"id" : "obj-45",
					"maxclass" : "number",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 277.0, 211.0, 53.0, 23.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"fontname" : "Arial",
					"fontsize" : 13.0,
					"id" : "obj-3",
					"maxclass" : "number",
					"numinlets" : 1,
					"numoutlets" : 2,
					"outlettype" : [ "", "bang" ],
					"parameter_enable" : 0,
					"patching_rect" : [ 135.0, 211.0, 53.0, 23.0 ],
					"style" : ""
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-19",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 3,
					"outlettype" : [ "float", "float", "float" ],
					"patching_rect" : [ 231.5, 126.0, 111.0, 22.0 ],
					"style" : "",
					"text" : "unpack 0. 0. 0."
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-15",
					"maxclass" : "comment",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 603.0, 20.0, 249.0, 20.0 ],
					"style" : "",
					"text" : "receive data from processing on port 13000"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-9",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 0,
					"patching_rect" : [ 36.0, 130.0, 34.0, 22.0 ],
					"style" : "",
					"text" : "print"
				}

			}
, 			{
				"box" : 				{
					"id" : "obj-1",
					"maxclass" : "newobj",
					"numinlets" : 1,
					"numoutlets" : 1,
					"outlettype" : [ "" ],
					"patching_rect" : [ 231.5, 24.0, 111.0, 22.0 ],
					"style" : "",
					"text" : "udpreceive 13000"
				}

			}
 ],
		"lines" : [ 			{
				"patchline" : 				{
					"destination" : [ "obj-56", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"order" : 0,
					"source" : [ "obj-1", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-9", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 241.0, 48.0, 45.5, 48.0 ],
					"order" : 1,
					"source" : [ "obj-1", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-3", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 241.0, 198.0, 144.5, 198.0 ],
					"source" : [ "obj-19", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-45", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 287.0, 150.0, 286.5, 150.0 ],
					"source" : [ "obj-19", 1 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-53", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 333.0, 198.0, 424.0, 198.0 ],
					"source" : [ "obj-19", 2 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-22", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-2", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-7", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-22", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-22", 3 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 367.0, 432.0, 279.0, 432.0 ],
					"source" : [ "obj-25", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-22", 1 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-29", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-2", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-3", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-22", 2 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 284.416656, 432.0, 234.166672, 432.0 ],
					"source" : [ "obj-30", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-30", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-45", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-19", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-51", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-51", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"source" : [ "obj-56", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-5", 1 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 144.5, 630.0, 174.5, 630.0 ],
					"order" : 0,
					"source" : [ "obj-7", 0 ]
				}

			}
, 			{
				"patchline" : 				{
					"destination" : [ "obj-5", 0 ],
					"disabled" : 0,
					"hidden" : 0,
					"midpoints" : [ 144.5, 639.0, 148.5, 639.0 ],
					"order" : 1,
					"source" : [ "obj-7", 0 ]
				}

			}
 ],
		"dependency_cache" : [  ],
		"autosave" : 0
	}

}
 



7. Microsoft Kinect as an interface

Réflexions - Kid Tested, Dpt., 2012.

Traceland*, Thomas Sanchez Lengeling
"The Installation addresses the issue of “Big Data” and the constant production of data by individuals using technical devices, leaving a more and more constant trace of their actions and movements."

Starfield by Lab212 from MuDA on Vimeo.


Tutorials
Daniel Shiffman Kinect for Windows App
Shiffman Kinect & Processing Basics

Eine Kinect ist eine Kamera mit integriertem Infrarotsensor, die von Microsoft eigentlich für ihre XBOX-Spielekonsole entwickelt wurde. Sie ist in der Lage, räumliche Informationen zu verarbeiten und bspw. ein dreidimensionales Modell von SpielerInnen digital zu verarbeiten. Bald wurde sie daher in Hacks ausserhalb ihres eigentlichen Kontexts für interaktive Installationen oder Medienkunstprojekte verwendet.

Beispiel: Interaktive Installation mit der Kinect als Sensor

Um besonders nahe liegende Extremitäten zu erkennen kann man den das Graustufenbild der Kinect, das die z-Position abbildet, für eine Blob detection verwenden.

Zunächst sollte die OpenKinect-Library von Daniel Shiffman importiert werden. Mit den Funktionen


import org.openkinect.processing.*;

Dann wird ein Kinect-Objekt deklariert:


Kinect2 kinect2;

Initialize:


void setup() {
  kinect2 = new Kinect2(this);
  kinect2.initDevice();
}

Folgende Funktionen werden von der openkinect-Library zur Verfügung gestellt, um die verschieden Bildquellen der Kinect auszulesen und in weiterer Folge die entsprechenden Daten zu nutzen:

Function Output
getVideoImage() RGB-Bild
getDepthImage() 3D Graustufenbild
getIrImage() Infrarotbild

Beispiel: Kinect Cam-Bilder darstellen


//Kinect-Library von Daniel Shiffman

import org.openkinect.freenect.*;        // Die benötigten Kinect-Libraries
import org.openkinect.freenect2.*;       // von Daniel Shiffmann importieren
import org.openkinect.processing.*;

int xPos = 512;                          // Breite des Kinect-Bildes
int yPos = 424;                          // Höhe des Kinect-Bildes

Kinect2 kin;                             // Das Kinect-Objekt deklarieren

void setup() {
  size(1024, 848, P2D);
  kin = new Kinect2(this);
  //kin.initVideo();
  //kin.initDepth();
  //kin.initIR();
  //kin.initRegistered();
  kin.initDevice();                      // Kinect-Daten starten
}

void draw() {
  background(0);                        // Bild bei jedem Durchlauf löschen
  image(kin.getVideoImage(), 0, 0, kin.colorWidth*0.267, kin.colorHeight*0.267);
  image(kin.getDepthImage(), xPos, 0);
  image(kin.getIrImage(), 0, yPos);
  image(kin.getRegisteredImage(), xPos, yPos);
}

  

Using raw depth data for a point cloud

The getRawDepth()-function of Kinect2 gives us back depth-data as an Array of integers between 0 and 4500. At every frame for each Pixel a number is stored. So knowing that the resolution of Kinect`s depth-cam is X we can loop through this Array to retrieve depth-information (you could also use the functions depthWidth and depthHeight to get the resolution).

// Based on an Example by Daniel Shiffman

import org.openkinect.processing.*;


Kinect2 kinect2;          // Kinect Library object

float rotationRads = 3.1;  // Angle for rotation
int[] depth;
int skip = 1;
int nearOffset = 200;
int distantOffset = 1200;

PImage colors;

void setup() {
  size(1400, 600, P3D);
  colorMode(HSB);
  kinect2 = new Kinect2(this);
  kinect2.initVideo();
  kinect2.initDepth();
  kinect2.initIR();
  kinect2.initRegistered();
  kinect2.initDevice();
}


void draw() {
  background(0);

  translate(width/2, height/2, -100);       // Translate and rotate
  rotateY(rotationRads);
  
  // Get the raw depth as array of integers
  depth = kinect2.getRawDepth();
  colors = kinect2.getDepthImage();
  
  
  // draw point cloud

  beginShape(POINTS);
  for (int x = 0; x < kinect2.depthWidth; x += skip) {
    for (int y = 0; y < kinect2.depthHeight; y += skip) {
      int offset = x + y * kinect2.depthWidth;
      if (depth[offset] < distantOffset && depth[offset] > nearOffset){ 
        PVector point = depthToPointCloudPos(x, y, depth[offset]);     //calculte the x, y, z camera position based on the raw depth data
        float bright = map(depth[offset], 1200, 600, 0, 255);
        stroke(240,255,255, bright);
        pushMatrix();
        translate(point.x, point.y, point.z-800);  
        vertex(point.x, point.y, point.z-800);                             // Draw a point
        popMatrix();  
     } 
    }
  }
  endShape();

  // Rotate
  //rotationRads += 0.005;
}


//calculate the xyz camera position based on the depth data
PVector depthToPointCloudPos(int x, int y, float depthValue) {
  PVector point = new PVector();
  point.z = (depthValue);// / (1.0f); // Convert from mm to meters
  point.x = (x - CameraParams.cx) * point.z / CameraParams.fx;
  point.y = (y - CameraParams.cy) * point.z / CameraParams.fy;
  return point;
}
  
//camera information based on the Kinect v2 hardware
static class CameraParams {
  static float cx = 254.878f;
  static float cy = 205.395f;
  static float fx = 365.456f;
  static float fy = 365.456f;
  static float k1 = 0.0905474;
  static float k2 = -0.26819;
  static float k3 = 0.0950862;
  static float p1 = 0.0;
  static float p2 = 0.0;
}
  

Threshold

If we want to single out objects at a certain distance, we can implement a threshold that allows only points of a specific range of depth-values.

Exercise (30 minutes):
Come up with an alternative visualization of the human body. Use the same points but translate them into different structures (e.g. connecting them, using them as coordinates for more complex objects, intersecting these objects, …)



Finding the contour of an object

Now we combine Kinect`s depth information with contour detection of the openCV-Library (see the first part of this lecture for a brief introduction to openCV)
It`s even easier to use the library blobDetection by Julien Gachadoat.

import blobDetection.*;
import gab.opencv.*;
import org.openkinect.processing.*;

BlobDetection theBlobDetection;
Kinect2 kinect2; 

PImage BWdepth;

void setup(){
  size(640, 480);
  kinect2 = new Kinect2(this);
  kinect2.initVideo();
  kinect2.initDepth();
  kinect2.initIR();
  kinect2.initRegistered();
  kinect2.initDevice();
  
  BWdepth = kinect2.getDepthImage();
  
  theBlobDetection = new BlobDetection(BWdepth.width, BWdepth.height);
  theBlobDetection.setPosDiscrimination(false);
  theBlobDetection.setThreshold(0.1);
  theBlobDetection.computeBlobs(BWdepth.pixels);
}

void draw(){
  BWdepth = kinect2.getDepthImage();
  theBlobDetection.computeBlobs(BWdepth.pixels);
  image(BWdepth, 0, 0, width, height);
  drawBlobsAndEdges();
}


void drawBlobsAndEdges() {
  Blob b;
  EdgeVertex eA, eB;
  for (int n=0; n < theBlobDetection.getBlobNb(); n++){   // loop through all found blobs
    b = theBlobDetection.getBlob(n);
    float blobWidth = b.w;                                // get width and height of current blob
    float blobHeight = b.h;
    if (b!=null && blobWidth > 0.4 && blobHeight > 0.6){  // if it`s a huge blob draw its contours
        strokeWeight(3);
        stroke(0, 255, 0);
        fill(255,0,0);       
        //PShape bs;
        //bs = createShape();
        //bs.beginShape();
        //bs.fill(0, 0, 255);  
        for (int m=0; m < b.getEdgeNb(); m++){              // loop through all vertices of current blob
          eA = b.getEdgeVertexA(m);
          eB = b.getEdgeVertexB(m);
          
          if (eA !=null && eB !=null)
           //bs.vertex(eA.x*width, eA.y*height);
           //bs.vertex(eB.x*width, eB.y*height);
            line(eA.x*width, eA.y*height, eB.x*width, eB.y*height);
        
         //bs.endShape();
         //shape(bs, 0, 0);
        }
    }
  }
}


How could the algorithm be impoved in a way to detect a body like in Marko Cicilianis piece "Steina"?


To do: