5. Sound, Visuals & Controller
Interaction, Controller, HyperInstruments
- Interaction between physical instrument and software
- expanding of sound through software
- software-instruments
- (generative) Realtime-scores by software
Example: Karlheinz Essl, Pachinko for toy piano and computer, 2013, http://www.essl.at/works/pachinko.html
Game Controller (e.g. XBox One Controller)
Download the Driver for OSX here (don`t get irritated – the "360"-driver will work).
Library for Processing: Game Control Plus. Or load it from the Contribution Manager of Processing.
Video Tutorial on how to configure a contoller for developers is here.
Create a configuration file for your controller
As a first step you will want to create a configuration file for your controller. We use the sketch provided by the library in Documents/Processing/libraries/GameControlPlus/examples/Configurator.
After you created the mapping you have throw the config-file into the data-folders of your upcoming sketches.
Examples
import net.java.games.input.*;
import org.gamecontrolplus.*;
import org.gamecontrolplus.gui.*;
ControlIO control;
ControlDevice stick;
float lx, ly, rx, ry, b1, b2;
void setup(){
size(800,600);
colorMode(HSB);
background(0);
fill(0,20);
stroke(0,0,0,10);
control = ControlIO.getInstance(this);
stick = control.getMatchedDevice("XBOXbasic");
if (stick == null){
println("nothing found");
System.exit(-1);
}
}
void getControllerInput(){
lx = map(stick.getSlider("STICK1X").getValue(),-1,1,0, width);
ly = map(stick.getSlider("STICK1Y").getValue(),-1,1,0, height);
rx = map(stick.getSlider("STICK2X").getValue(),-1,1,40, 80);
ry = map(stick.getSlider("STICK2Y").getValue(),-1,1,0, 255);
b1 = map(stick.getSlider("BUTTON1").getValue(),-1,1,-1, 1);
b2 = map(stick.getSlider("BUTTON2").getValue(),-1,1,-1, 1);
println(rx);
}
void draw(){
getControllerInput();
stroke(rx, 100, 255, 10);
line(0,0, lx, ly);
line(width, 0, lx, ly);
line(0,height, lx, ly);
line(width,height, lx, ly);
}
Exercise (20 minutes):
Come up with your own visualization. Use data from both sticks and from a button.
Leap Motion Detection
Beispiel: Leap Motion hand recognition for animation
Beispiel
Motion-controlled Servos with Leap Motion & Raspberry Pi from PubNub on Vimeo.
Leap Motion Music Visualizer from mtytel on Vimeo.
Leap Motion in Processing
Library Leap Motion for Processing installieren.
The documentation of the API provided by Leap Motion is accessible in the developer sectionof their website.
A tutorial explaining the use of Leap Motion can be found here.
Leap Motion offers a huge amount of RAW data that can be used for further individual processing.
Example: Using Leap Motion
import de.voidplus.leapmotion.*;
LeapMotion leap;
void setup() {
size(800, 500, OPENGL);
leap = new LeapMotion(this);
}
void draw() {
background(255);
int fps = leap.getFrameRate();
text(fps + " fps", 20, 20);
for (Hand hand : leap.getHands ()) {
hand.draw();
}
}
Example: Hand-Painting with Leap Motion
import de.voidplus.leapmotion.*;
LeapMotion leap;
float INTERACTION_SPACE_WIDTH = 200; // left-right from user
float INTERACTION_SPACE_DEPTH = 150; // away-and-toward user
float colorShift = 0;
float thbDistance, idxDistance, mdlDistance, rngDistance, pkyDistance;
void setup() {
size(690, 500, P3D);
leap = new LeapMotion(this);
colorMode(HSB, 255);
background(0);
}
void draw() {
noStroke();
fill(colorShift,255,30, 2);
rect(0,0,width,height);
if (colorShift < 215){
colorShift += 1.5;
}else{
colorShift = 0;
}
int fps = leap.getFrameRate();
for (Hand hand : leap.getHands ()) {
PVector handPos = hand.getRawPosition();
PVector thumbTip = hand.getThumb().getRawPositionOfJointTip();
PVector indexTip = hand.getIndexFinger().getRawPositionOfJointTip();
PVector middleTip = hand.getMiddleFinger().getRawPositionOfJointTip();
PVector ringTip = hand.getRingFinger().getRawPositionOfJointTip();
PVector pinkyTip = hand.getPinkyFinger().getRawPositionOfJointTip();
thbDistance = PVector.dist(handPos, thumbTip); // Calculate the distance of fingertips to palm
idxDistance = PVector.dist(handPos, indexTip);
mdlDistance = PVector.dist(handPos, middleTip);
rngDistance = PVector.dist(handPos, ringTip);
pkyDistance = PVector.dist(handPos, pinkyTip);
println(thbDistance);
handleFinger(thumbTip, "thb");
handleFinger(indexTip, "idx");
handleFinger(middleTip, "mdl");
handleFinger(ringTip, "rng");
handleFinger(pinkyTip, "pky");
}
}
void handleFinger(PVector pos, String id) {
float radius = 0;
// map finger tip position to 2D surface
float x = map(pos.x, -INTERACTION_SPACE_WIDTH,
INTERACTION_SPACE_WIDTH, 0, width);
float y = map(pos.z, -INTERACTION_SPACE_DEPTH,
INTERACTION_SPACE_DEPTH, 0, height);
colorMode(HSB, 255);
if (id == "thb"){
fill(0+colorShift,100,255,20);
radius = pow(thbDistance,3)/10000;
}else if (id == "idx"){
fill(10+colorShift,100,255,20);
radius = pow(idxDistance,3)/10000;
}else if (id == "mdl"){
fill(20+colorShift,100,255,20);
radius = pow(mdlDistance,3)/10000;
}else if (id == "rng"){
fill(30+colorShift,100,255,20);
radius = pow(rngDistance,3)/10000;
}else if (id == "pky"){
fill(40+colorShift,100,255,20);
radius = pow(pkyDistance,3)/10000;
}
noStroke();
ellipse(x, y, radius, radius);
}
Leap Motion is also capable of detection some gestures. See the example in the library`s folder!
Processing with MAX
Mew, Emily Groves
Creating instruments in MAX
The rather complex synthesizer ›X.FM‹ comes with MAX7
Exercise (20 minutes):
Build your own tiny MAX-instrument. Use the objects saw~, etc.
Exchange data with Open Sound Control (OSC)
"Open Sound Control (OSC) is a protocol for communication among computers, sound synthesizers, and other multimedia devices that is optimized for modern networking technology. Bringing the benefits of modern networking technology to the world of electronic musical instruments, OSC's advantages include interoperability, accuracy, flexibility, and enhanced organization and documentation." (opensoundcontrol.org)
Exchange Data between Processing and MAX
MAX patch:
import netP5.*;
import oscP5.*;
OscP5 osc;
NetAddress myRemoteLocation;
void setup(){
size(800, 600);
osc = new OscP5(this, 12000);
myRemoteLocation = new NetAddress("127.0.0.1", 13000);
}
void draw(){
}
void mousePressed(){
OscMessage myMessage = new OscMessage("hello");
myMessage.add("world!");
osc.send(myMessage, myRemoteLocation);
println(myMessage);
}
// Libs for contraoller
import net.java.games.input.*;
import org.gamecontrolplus.*;
import org.gamecontrolplus.gui.*;
// Libs for OSC communication with MAX
import netP5.*;
import oscP5.*;
ControlIO control;
ControlDevice stick;
OscP5 osc;
NetAddress myRemoteLocation;
float lx, ly, rx, ry, b1, b2;
float x,y;
boolean b = false;
PVector Walker;
PVector SourceOne, SourceTwo, SourceThree;
float distOne, distTwo, distThree;
void setup() {
size(1280, 720, P3D);
colorMode(HSB);
control = ControlIO.getInstance(this);
stick = control.getMatchedDevice("XBOXbasic");
osc = new OscP5(this, 12000);
myRemoteLocation = new NetAddress("127.0.0.1", 13000);
SourceOne = new PVector(40, 380);
SourceTwo = new PVector(600, 200);
SourceThree = new PVector(900, 550);
Walker = new PVector(width/2, height/2);
}
void draw() {
// 1. get controller data input
getControllerInput();
// 2. do the (simple) algorithms
Walker.set(lx, ly); // control walker with left stick
distOne = Walker.dist(SourceOne);
distTwo = Walker.dist(SourceTwo);
distThree = Walker.dist(SourceThree);
// 4. map the data to the parameters of ellipses and do the visuals
background(0);
ellipse(Walker.x,Walker.y,10,10);
stroke(0, 0,255, 100);
fill(20,255,255,50);
ellipse(SourceOne.x, SourceOne.y, 2*distOne,2*distOne);
fill(35,255,255,50);
ellipse(SourceTwo.x, SourceTwo.y, 2*distTwo, 2*distTwo);
fill(50,255,255,50);
ellipse(SourceThree.x, SourceThree.y, 2*distThree, 2*distThree);
stroke(0);
fill(0,0,255);
ellipse(Walker.x,Walker.y,10,10);
// send OSC to MAX
OscMessage myMessage = new OscMessage("");
myMessage.add(Walker.x+" "+Walker.y+" "+distOne+" "+distTwo+" "+distThree);
osc.send(myMessage, myRemoteLocation);
}
void getControllerInput(){
lx = map(stick.getSlider("STICK1X").getValue(),-1,1,0, width);
ly = map(stick.getSlider("STICK1Y").getValue(),-1,1,0, height);
rx = map(stick.getSlider("STICK2X").getValue(),-1,1,40, 80);
ry = map(stick.getSlider("STICK2Y").getValue(),-1,1,0, 255);
b1 = map(stick.getSlider("BUTTON1").getValue(),-1,1,-1, 1);
b2 = map(stick.getSlider("BUTTON2").getValue(),-1,1,-1, 1);
}
{
"patcher" : {
"fileversion" : 1,
"appversion" : {
"major" : 7,
"minor" : 3,
"revision" : 3,
"architecture" : "x86",
"modernui" : 1
}
,
"rect" : [ 303.0, 225.0, 993.0, 893.0 ],
"bglocked" : 0,
"openinpresentation" : 0,
"default_fontsize" : 12.0,
"default_fontface" : 0,
"default_fontname" : "Arial",
"gridonopen" : 1,
"gridsize" : [ 15.0, 15.0 ],
"gridsnaponopen" : 1,
"objectsnaponopen" : 1,
"statusbarvisible" : 2,
"toolbarvisible" : 1,
"lefttoolbarpinned" : 0,
"toptoolbarpinned" : 0,
"righttoolbarpinned" : 0,
"bottomtoolbarpinned" : 0,
"toolbars_unpinned_last_save" : 0,
"tallnewobj" : 0,
"boxanimatetime" : 200,
"enablehscroll" : 1,
"enablevscroll" : 1,
"devicewidth" : 0.0,
"description" : "",
"digest" : "",
"tags" : "",
"style" : "",
"subpatcher_template" : "",
"boxes" : [ {
"box" : {
"format" : 6,
"id" : "obj-69",
"maxclass" : "flonum",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 825.0, 478.0, 50.0, 22.0 ],
"style" : ""
}
}
, {
"box" : {
"format" : 6,
"id" : "obj-68",
"maxclass" : "flonum",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 496.5, 446.5, 50.0, 22.0 ],
"style" : ""
}
}
, {
"box" : {
"fontname" : "Arial",
"fontsize" : 13.0,
"id" : "obj-64",
"maxclass" : "message",
"numinlets" : 2,
"numoutlets" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 61.0, 665.0, 43.0, 23.0 ],
"style" : "",
"text" : "clear"
}
}
, {
"box" : {
"attr" : "edit_mode",
"id" : "obj-63",
"maxclass" : "attrui",
"numinlets" : 1,
"numoutlets" : 1,
"orientation" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 610.0, 459.0, 75.0, 44.0 ],
"presentation_rect" : [ 0.0, 0.0, 0.0, 44.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-62",
"maxclass" : "newobj",
"numinlets" : 6,
"numoutlets" : 1,
"outlettype" : [ "signal" ],
"patching_rect" : [ 177.5, 707.0, 71.5, 22.0 ],
"style" : "",
"text" : "biquad~"
}
}
, {
"box" : {
"id" : "obj-61",
"maxclass" : "newobj",
"numinlets" : 2,
"numoutlets" : 1,
"outlettype" : [ "int" ],
"patching_rect" : [ 511.5, 318.0, 56.5, 22.0 ],
"style" : "",
"text" : "* 5"
}
}
, {
"box" : {
"fontface" : 0,
"id" : "obj-60",
"maxclass" : "filtergraph~",
"nfilters" : 1,
"numinlets" : 8,
"numoutlets" : 7,
"outlettype" : [ "list", "float", "float", "float", "float", "list", "int" ],
"parameter_enable" : 0,
"patching_rect" : [ 590.0, 513.0, 254.0, 136.0 ],
"setfilter" : [ 0, 3, 1, 0, 0, 1485.0, 3.540734, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-58",
"maxclass" : "newobj",
"numinlets" : 2,
"numoutlets" : 1,
"outlettype" : [ "int" ],
"patching_rect" : [ 371.5, 361.0, 29.5, 22.0 ],
"style" : "",
"text" : "/ 5"
}
}
, {
"box" : {
"id" : "obj-57",
"maxclass" : "newobj",
"numinlets" : 2,
"numoutlets" : 1,
"outlettype" : [ "int" ],
"patching_rect" : [ 441.5, 361.0, 53.0, 22.0 ],
"style" : "",
"text" : "/ 10"
}
}
, {
"box" : {
"id" : "obj-55",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 811.5, 235.0, 161.0, 20.0 ],
"presentation_rect" : [ 811.5, 235.0, 0.0, 0.0 ],
"style" : "",
"text" : "data from processing"
}
}
, {
"box" : {
"id" : "obj-54",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 795.5, 813.0, 161.0, 20.0 ],
"presentation_rect" : [ 795.5, 813.0, 0.0, 0.0 ],
"style" : "",
"text" : "output"
}
}
, {
"box" : {
"id" : "obj-53",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 811.5, 421.0, 161.0, 20.0 ],
"presentation_rect" : [ 811.5, 420.0, 0.0, 0.0 ],
"style" : "",
"text" : "mapping"
}
}
, {
"box" : {
"id" : "obj-52",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 811.5, 318.0, 161.0, 20.0 ],
"style" : "",
"text" : "calculations"
}
}
, {
"box" : {
"id" : "obj-50",
"maxclass" : "newobj",
"numinlets" : 2,
"numoutlets" : 1,
"outlettype" : [ "int" ],
"patching_rect" : [ 441.5, 318.0, 53.0, 22.0 ],
"style" : "",
"text" : "- 400"
}
}
, {
"box" : {
"id" : "obj-49",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 511.5, 274.0, 79.0, 20.0 ],
"presentation_rect" : [ 683.5, 274.0, 0.0, 0.0 ],
"style" : "",
"text" : "distThree"
}
}
, {
"box" : {
"id" : "obj-48",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 441.5, 274.0, 53.0, 20.0 ],
"presentation_rect" : [ 612.5, 274.0, 0.0, 0.0 ],
"style" : "",
"text" : "distTwo"
}
}
, {
"box" : {
"id" : "obj-47",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 371.5, 274.0, 53.0, 20.0 ],
"presentation_rect" : [ 541.0, 274.0, 0.0, 0.0 ],
"style" : "",
"text" : "distOne"
}
}
, {
"box" : {
"id" : "obj-46",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 288.0, 274.0, 53.0, 20.0 ],
"presentation_rect" : [ 457.0, 274.0, 0.0, 0.0 ],
"style" : "",
"text" : "walkerY"
}
}
, {
"box" : {
"id" : "obj-44",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 196.0, 274.0, 53.0, 20.0 ],
"style" : "",
"text" : "walkerX"
}
}
, {
"box" : {
"id" : "obj-43",
"linecount" : 2,
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 38.0, 212.0, 112.0, 33.0 ],
"style" : "",
"text" : "print received data to console"
}
}
, {
"box" : {
"id" : "obj-42",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 548.0, 437.0, 125.5, 20.0 ],
"presentation_rect" : [ 689.0, 350.0, 0.0, 0.0 ],
"style" : "",
"text" : "minimum loop point"
}
}
, {
"box" : {
"id" : "obj-41",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 427.5, 437.0, 112.0, 20.0 ],
"presentation_rect" : [ 571.5, 350.0, 0.0, 0.0 ],
"style" : "",
"text" : "playback speed"
}
}
, {
"box" : {
"id" : "obj-40",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 301.0, 437.0, 112.0, 20.0 ],
"presentation_rect" : [ 446.0, 350.0, 0.0, 0.0 ],
"style" : "",
"text" : "Start at X ms"
}
}
, {
"box" : {
"format" : 6,
"id" : "obj-39",
"maxclass" : "flonum",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 548.0, 464.0, 50.0, 22.0 ],
"style" : ""
}
}
, {
"box" : {
"format" : 6,
"id" : "obj-38",
"maxclass" : "flonum",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 427.5, 464.0, 50.0, 22.0 ],
"style" : ""
}
}
, {
"box" : {
"format" : 6,
"id" : "obj-34",
"maxclass" : "flonum",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 301.0, 464.0, 50.0, 22.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-33",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 1,
"outlettype" : [ "signal" ],
"patching_rect" : [ 427.5, 513.0, 33.0, 22.0 ],
"style" : "",
"text" : "sig~"
}
}
, {
"box" : {
"id" : "obj-32",
"maxclass" : "message",
"numinlets" : 2,
"numoutlets" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 168.5, 464.0, 43.0, 22.0 ],
"style" : "",
"text" : "loop 1"
}
}
, {
"box" : {
"id" : "obj-31",
"maxclass" : "gain~",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "signal", "int" ],
"parameter_enable" : 0,
"patching_rect" : [ 168.5, 753.0, 119.0, 40.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-27",
"maxclass" : "newobj",
"numinlets" : 3,
"numoutlets" : 2,
"outlettype" : [ "signal", "signal" ],
"patching_rect" : [ 177.5, 665.0, 119.0, 22.0 ],
"saved_object_attributes" : {
"basictuning" : 440,
"followglobaltempo" : 0,
"formantcorrection" : 0,
"loopend" : [ 0.0, "ms" ],
"loopstart" : [ 681.0, "ms" ],
"mode" : "basic",
"originallength" : [ 0.0, "ticks" ],
"originaltempo" : 120.0,
"phase" : [ 0.0, "ticks" ],
"pitchcorrection" : 0,
"quality" : "basic",
"timestretch" : [ 0 ]
}
,
"style" : "",
"text" : "groove~ loopsample"
}
}
, {
"box" : {
"id" : "obj-26",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 610.0, 161.0, 150.0, 20.0 ],
"presentation_rect" : [ 32.0, 354.0, 0.0, 0.0 ],
"style" : "",
"text" : "load sample",
"textjustification" : 0
}
}
, {
"box" : {
"id" : "obj-25",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 168.5, 437.0, 112.0, 20.0 ],
"style" : "",
"text" : "turn looping on",
"textjustification" : 0
}
}
, {
"box" : {
"fontname" : "Arial",
"fontsize" : 13.0,
"id" : "obj-16",
"maxclass" : "number",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 511.5, 235.0, 53.0, 23.0 ],
"style" : ""
}
}
, {
"box" : {
"fontname" : "Arial",
"fontsize" : 13.0,
"id" : "obj-15",
"maxclass" : "number",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 441.5, 235.0, 53.0, 23.0 ],
"style" : ""
}
}
, {
"box" : {
"fontname" : "Arial",
"fontsize" : 13.0,
"id" : "obj-14",
"maxclass" : "number",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 371.5, 235.0, 53.0, 23.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-13",
"maxclass" : "message",
"numinlets" : 2,
"numoutlets" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 610.0, 69.0, 162.0, 22.0 ],
"style" : "",
"text" : "read"
}
}
, {
"box" : {
"id" : "obj-56",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 321.0, 100.0, 116.0, 22.0 ],
"style" : "",
"text" : "tosymbol"
}
}
, {
"box" : {
"id" : "obj-51",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 321.0, 133.0, 116.0, 22.0 ],
"style" : "",
"text" : "fromsymbol"
}
}
, {
"box" : {
"fontname" : "Arial",
"fontsize" : 13.0,
"id" : "obj-45",
"maxclass" : "number",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 288.0, 235.0, 53.0, 23.0 ],
"style" : ""
}
}
, {
"box" : {
"fontname" : "Arial",
"fontsize" : 13.0,
"id" : "obj-3",
"maxclass" : "number",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 196.0, 235.0, 53.0, 23.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-19",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 5,
"outlettype" : [ "float", "float", "float", "float", "float" ],
"patching_rect" : [ 321.0, 167.0, 116.0, 22.0 ],
"style" : "",
"text" : "unpack 0. 0. 0. 0. 0."
}
}
, {
"box" : {
"id" : "obj-9",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 42.0, 167.0, 34.0, 22.0 ],
"style" : "",
"text" : "print"
}
}
, {
"box" : {
"id" : "obj-12",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 321.0, 66.0, 116.0, 22.0 ],
"style" : "",
"text" : "udpreceive 13000"
}
}
, {
"box" : {
"id" : "obj-4",
"maxclass" : "ezdac~",
"numinlets" : 2,
"numoutlets" : 0,
"patching_rect" : [ 168.5, 824.0, 45.0, 45.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-1",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "float", "bang" ],
"patching_rect" : [ 610.0, 123.0, 162.0, 22.0 ],
"style" : "",
"text" : "buffer~ loopsample anton.aif"
}
}
],
"lines" : [ {
"patchline" : {
"destination" : [ "obj-56", 0 ],
"disabled" : 0,
"hidden" : 0,
"order" : 0,
"source" : [ "obj-12", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-9", 0 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 330.5, 90.0, 51.5, 90.0 ],
"order" : 1,
"source" : [ "obj-12", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-1", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-13", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-58", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-14", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-50", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-15", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-61", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-16", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-14", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-19", 2 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-15", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-19", 3 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-16", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-19", 4 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-3", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-19", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-45", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-19", 1 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-62", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-27", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-4", 1 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 278.0, 810.0, 204.0, 810.0 ],
"source" : [ "obj-31", 1 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-4", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-31", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-27", 0 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 178.0, 651.0, 187.0, 651.0 ],
"source" : [ "obj-32", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-27", 0 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 437.0, 651.0, 187.0, 651.0 ],
"source" : [ "obj-33", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-27", 0 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 310.5, 651.0, 187.0, 651.0 ],
"source" : [ "obj-34", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-33", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-38", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-27", 1 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 557.5, 651.0, 237.0, 651.0 ],
"source" : [ "obj-39", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-57", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-50", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-19", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-51", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-51", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-56", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-38", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-57", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-31", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-58", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-62", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-60", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-60", 5 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 521.0, 423.0, 767.357117, 423.0 ],
"source" : [ "obj-61", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-31", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-62", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-60", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-63", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-62", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-64", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-60", 7 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-69", 0 ]
}
}
],
"parameters" : {
}
,
"dependency_cache" : [ ],
"autosave" : 0
}
}
import de.voidplus.leapmotion.*;
import netP5.*;
import oscP5.*;
OscP5 osc;
NetAddress myRemoteLocation;
LeapMotion leap;
ArrayList haende = new ArrayList();
ArrayList indexPoints = new ArrayList();
float distance;
void setup() {
size(690, 500, P3D);
osc = new OscP5(this, 12000);
myRemoteLocation = new NetAddress("127.0.0.1", 13000);
leap = new LeapMotion(this);
colorMode(HSB, 255);
noStroke();
fill(50,255,255);
lights();
indexPoints.add(new PVector(0,0,0)); // add Vector for left index finger
indexPoints.add(new PVector(0,0,0)); // add Vector for right index finger
}
void draw() {
background(255);
// get all detected hands an put them in an ArrayList
ArrayList hands = leap.getHands();
// loop thorugh all detected hand-objects
for (int i = 0; i < hands.size() ; i++){
Hand currentHand = hands.get(i);
PVector indexTip = currentHand.getIndexFinger().getRawPositionOfJointTip();
PVector currentTip = indexPoints.get(i);
currentTip.set(indexTip);
// draw spheres at positions of index fingers
noStroke();
pushMatrix();
translate(width/2, 0, 0);
translate(indexTip.x, height-indexTip.y, indexTip.z);
sphere(currentTip.y/10);
sphere(20);
popMatrix();
// Send Data to MAX
OscMessage myMessage = new OscMessage("");
myMessage.add(distance+" "+currentTip.y);
osc.send(myMessage, myRemoteLocation);
}
// Calculate distance between index fingers and draw the line
if (indexPoints.size() == 2){
println(indexPoints.size());
PVector Pleft = indexPoints.get(0);
PVector Pright = indexPoints.get(1);
distance = Pleft.dist(Pright);
stroke(0,0,0);
strokeWeight((1000/distance));
translate(width/2, 0, 0);
line(Pleft.x, height-Pleft.y, Pleft.z, Pright.x, height-Pright.y, Pright.z);
}
}
{
"patcher" : {
"fileversion" : 1,
"appversion" : {
"major" : 7,
"minor" : 3,
"revision" : 3,
"architecture" : "x86",
"modernui" : 1
}
,
"rect" : [ 451.0, 344.0, 918.0, 734.0 ],
"bglocked" : 0,
"openinpresentation" : 0,
"default_fontsize" : 12.0,
"default_fontface" : 0,
"default_fontname" : "Arial",
"gridonopen" : 1,
"gridsize" : [ 15.0, 15.0 ],
"gridsnaponopen" : 1,
"objectsnaponopen" : 1,
"statusbarvisible" : 2,
"toolbarvisible" : 1,
"lefttoolbarpinned" : 0,
"toptoolbarpinned" : 0,
"righttoolbarpinned" : 0,
"bottomtoolbarpinned" : 0,
"toolbars_unpinned_last_save" : 0,
"tallnewobj" : 0,
"boxanimatetime" : 200,
"enablehscroll" : 1,
"enablevscroll" : 1,
"devicewidth" : 0.0,
"description" : "",
"digest" : "",
"tags" : "",
"style" : "",
"subpatcher_template" : "",
"boxes" : [ {
"box" : {
"format" : 6,
"id" : "obj-30",
"maxclass" : "flonum",
"minimum" : 0.0,
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 274.916656, 385.0, 50.0, 22.0 ],
"style" : ""
}
}
, {
"box" : {
"format" : 6,
"id" : "obj-29",
"maxclass" : "flonum",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 179.833328, 383.0, 35.0, 22.0 ],
"presentation_rect" : [ 206.5, 385.0, 0.0, 0.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-28",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 179.833328, 360.0, 53.0, 20.0 ],
"presentation_rect" : [ 209.5, 360.0, 0.0, 0.0 ],
"style" : "",
"text" : "gain"
}
}
, {
"box" : {
"id" : "obj-27",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 274.916656, 360.0, 69.0, 20.0 ],
"presentation_rect" : [ 264.5, 354.0, 0.0, 0.0 ],
"style" : "",
"text" : "centerfrq"
}
}
, {
"box" : {
"id" : "obj-26",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 357.5, 360.0, 53.0, 20.0 ],
"presentation_rect" : [ 322.0, 354.0, 0.0, 0.0 ],
"style" : "",
"text" : "Q"
}
}
, {
"box" : {
"format" : 6,
"id" : "obj-25",
"maxclass" : "flonum",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 357.5, 385.0, 50.0, 22.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-22",
"maxclass" : "newobj",
"numinlets" : 4,
"numoutlets" : 1,
"outlettype" : [ "signal" ],
"patching_rect" : [ 135.0, 446.0, 153.5, 22.0 ],
"style" : "",
"text" : "reson~"
}
}
, {
"box" : {
"id" : "obj-10",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 414.5, 249.0, 53.0, 20.0 ],
"presentation_rect" : [ 275.0, 232.0, 0.0, 0.0 ],
"style" : "",
"text" : "rightY"
}
}
, {
"box" : {
"id" : "obj-6",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 277.0, 249.0, 53.0, 20.0 ],
"style" : "",
"text" : "leftY"
}
}
, {
"box" : {
"id" : "obj-44",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 135.0, 247.0, 53.0, 20.0 ],
"style" : "",
"text" : "distance"
}
}
, {
"box" : {
"id" : "obj-7",
"maxclass" : "gain~",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "signal", "int" ],
"parameter_enable" : 0,
"patching_rect" : [ 135.0, 519.0, 111.0, 37.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-5",
"maxclass" : "ezdac~",
"numinlets" : 2,
"numoutlets" : 0,
"patching_rect" : [ 139.0, 643.0, 45.0, 45.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-2",
"maxclass" : "newobj",
"numinlets" : 2,
"numoutlets" : 1,
"outlettype" : [ "signal" ],
"patching_rect" : [ 135.0, 383.0, 39.0, 22.0 ],
"style" : "",
"text" : "saw~"
}
}
, {
"box" : {
"id" : "obj-56",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 231.5, 63.0, 111.0, 22.0 ],
"style" : "",
"text" : "tosymbol"
}
}
, {
"box" : {
"fontname" : "Arial",
"fontsize" : 13.0,
"id" : "obj-53",
"maxclass" : "number",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 414.5, 211.0, 53.0, 23.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-52",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 603.0, 122.0, 249.0, 20.0 ],
"style" : "",
"text" : "covert to numbers"
}
}
, {
"box" : {
"id" : "obj-51",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 231.5, 93.0, 111.0, 22.0 ],
"style" : "",
"text" : "fromsymbol"
}
}
, {
"box" : {
"fontname" : "Arial",
"fontsize" : 13.0,
"id" : "obj-45",
"maxclass" : "number",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 277.0, 211.0, 53.0, 23.0 ],
"style" : ""
}
}
, {
"box" : {
"fontname" : "Arial",
"fontsize" : 13.0,
"id" : "obj-3",
"maxclass" : "number",
"numinlets" : 1,
"numoutlets" : 2,
"outlettype" : [ "", "bang" ],
"parameter_enable" : 0,
"patching_rect" : [ 135.0, 211.0, 53.0, 23.0 ],
"style" : ""
}
}
, {
"box" : {
"id" : "obj-19",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 3,
"outlettype" : [ "float", "float", "float" ],
"patching_rect" : [ 231.5, 126.0, 111.0, 22.0 ],
"style" : "",
"text" : "unpack 0. 0. 0."
}
}
, {
"box" : {
"id" : "obj-15",
"maxclass" : "comment",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 603.0, 20.0, 249.0, 20.0 ],
"style" : "",
"text" : "receive data from processing on port 13000"
}
}
, {
"box" : {
"id" : "obj-9",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 0,
"patching_rect" : [ 36.0, 130.0, 34.0, 22.0 ],
"style" : "",
"text" : "print"
}
}
, {
"box" : {
"id" : "obj-1",
"maxclass" : "newobj",
"numinlets" : 1,
"numoutlets" : 1,
"outlettype" : [ "" ],
"patching_rect" : [ 231.5, 24.0, 111.0, 22.0 ],
"style" : "",
"text" : "udpreceive 13000"
}
}
],
"lines" : [ {
"patchline" : {
"destination" : [ "obj-56", 0 ],
"disabled" : 0,
"hidden" : 0,
"order" : 0,
"source" : [ "obj-1", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-9", 0 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 241.0, 48.0, 45.5, 48.0 ],
"order" : 1,
"source" : [ "obj-1", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-3", 0 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 241.0, 198.0, 144.5, 198.0 ],
"source" : [ "obj-19", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-45", 0 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 287.0, 150.0, 286.5, 150.0 ],
"source" : [ "obj-19", 1 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-53", 0 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 333.0, 198.0, 424.0, 198.0 ],
"source" : [ "obj-19", 2 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-22", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-2", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-7", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-22", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-22", 3 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 367.0, 432.0, 279.0, 432.0 ],
"source" : [ "obj-25", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-22", 1 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-29", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-2", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-3", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-22", 2 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 284.416656, 432.0, 234.166672, 432.0 ],
"source" : [ "obj-30", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-30", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-45", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-19", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-51", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-51", 0 ],
"disabled" : 0,
"hidden" : 0,
"source" : [ "obj-56", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-5", 1 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 144.5, 630.0, 174.5, 630.0 ],
"order" : 0,
"source" : [ "obj-7", 0 ]
}
}
, {
"patchline" : {
"destination" : [ "obj-5", 0 ],
"disabled" : 0,
"hidden" : 0,
"midpoints" : [ 144.5, 639.0, 148.5, 639.0 ],
"order" : 1,
"source" : [ "obj-7", 0 ]
}
}
],
"dependency_cache" : [ ],
"autosave" : 0
}
}
Kinect revisited
Réflexions - Kid Tested, Dpt., 2012.
Traceland*, Thomas Sanchez Lengeling
"The Installation addresses the issue of “Big Data” and the constant production of data by individuals using technical devices, leaving a more and more constant trace of their actions and movements."
Starfield by Lab212 from MuDA on Vimeo.
Tutorials
Daniel Shiffman Kinect for Windows App
Shiffman Kinect & Processing Basics
Eine Kinect ist eine Kamera mit integriertem Infrarotsensor, die von Microsoft eigentlich für ihre XBOX-Spielekonsole entwickelt wurde. Sie ist in der Lage, räumliche Informationen zu verarbeiten und bspw. ein dreidimensionales Modell von SpielerInnen digital zu verarbeiten. Bald wurde sie daher in Hacks ausserhalb ihres eigentlichen Kontexts für interaktive Installationen oder Medienkunstprojekte verwendet.
Beispiel: Interaktive Installation mit der Kinect als Sensor
Um besonders nahe liegende Extremitäten zu erkennen kann man den das Graustufenbild der Kinect, das die z-Position abbildet, für eine Blob detection verwenden.
Zunächst sollte die OpenKinect-Library von Daniel Shiffman importiert werden. Mit den Funktionen
import org.openkinect.processing.*;
Dann wird ein Kinect-Objekt deklariert:
Kinect2 kinect2;
Initialize:
void setup() {
kinect2 = new Kinect2(this);
kinect2.initDevice();
}
Folgende Funktionen werden von der openkinect-Library zur Verfügung gestellt, um die verschieden Bildquellen der Kinect auszulesen und in weiterer Folge die entsprechenden Daten zu nutzen:
| Function | Output |
|---|---|
| getVideoImage() | RGB-Bild |
| getDepthImage() | 3D Graustufenbild |
| getIrImage() | Infrarotbild |
Beispiel: Kinect Cam-Bilder darstellen
//Kinect-Library von Daniel Shiffman
import org.openkinect.freenect.*; // Die benötigten Kinect-Libraries
import org.openkinect.freenect2.*; // von Daniel Shiffmann importieren
import org.openkinect.processing.*;
int xPos = 512; // Breite des Kinect-Bildes
int yPos = 424; // Höhe des Kinect-Bildes
Kinect2 kin; // Das Kinect-Objekt deklarieren
void setup() {
size(1024, 848, P2D);
kin = new Kinect2(this);
//kin.initVideo();
//kin.initDepth();
//kin.initIR();
//kin.initRegistered();
kin.initDevice(); // Kinect-Daten starten
}
void draw() {
background(0); // Bild bei jedem Durchlauf löschen
image(kin.getVideoImage(), 0, 0, kin.colorWidth*0.267, kin.colorHeight*0.267);
image(kin.getDepthImage(), xPos, 0);
image(kin.getIrImage(), 0, yPos);
image(kin.getRegisteredImage(), xPos, yPos);
}
Using raw depth data for a point cloud
The getRawDepth()-function of Kinect2 gives us back depth-data as an Array of integers between 0 and 4500. At every frame for each Pixel a number is stored. So knowing that the resolution of Kinect`s depth-cam is X we can loop through this Array to retrieve depth-information (you could also use the functions depthWidth and depthHeight to get the resolution).
// Based on an Example by Daniel Shiffman
import org.openkinect.processing.*;
Kinect2 kinect2; // Kinect Library object
float rotationRads = 3.1; // Angle for rotation
int[] depth;
int skip = 1;
int nearOffset = 200;
int distantOffset = 1200;
PImage colors;
void setup() {
size(1400, 600, P3D);
colorMode(HSB);
kinect2 = new Kinect2(this);
kinect2.initVideo();
kinect2.initDepth();
kinect2.initIR();
kinect2.initRegistered();
kinect2.initDevice();
}
void draw() {
background(0);
translate(width/2, height/2, -100); // Translate and rotate
rotateY(rotationRads);
// Get the raw depth as array of integers
depth = kinect2.getRawDepth();
colors = kinect2.getDepthImage();
// draw point cloud
beginShape(POINTS);
for (int x = 0; x < kinect2.depthWidth; x += skip) {
for (int y = 0; y < kinect2.depthHeight; y += skip) {
int offset = x + y * kinect2.depthWidth;
if (depth[offset] < distantOffset && depth[offset] > nearOffset){
PVector point = depthToPointCloudPos(x, y, depth[offset]); //calculte the x, y, z camera position based on the raw depth data
float bright = map(depth[offset], 1200, 600, 0, 255);
stroke(240,255,255, bright);
pushMatrix();
translate(point.x, point.y, point.z-800);
vertex(point.x, point.y, point.z-800); // Draw a point
popMatrix();
}
}
}
endShape();
// Rotate
//rotationRads += 0.005;
}
//calculate the xyz camera position based on the depth data
PVector depthToPointCloudPos(int x, int y, float depthValue) {
PVector point = new PVector();
point.z = (depthValue);// / (1.0f); // Convert from mm to meters
point.x = (x - CameraParams.cx) * point.z / CameraParams.fx;
point.y = (y - CameraParams.cy) * point.z / CameraParams.fy;
return point;
}
//camera information based on the Kinect v2 hardware
static class CameraParams {
static float cx = 254.878f;
static float cy = 205.395f;
static float fx = 365.456f;
static float fy = 365.456f;
static float k1 = 0.0905474;
static float k2 = -0.26819;
static float k3 = 0.0950862;
static float p1 = 0.0;
static float p2 = 0.0;
}
Threshold
If we want to single out objects at a certain distance, we can implement a threshold that allows only points of a specific range of depth-values.
Exercise (30 minutes):
Come up with an alternative visualization of the human body. Use the same points but translate them into different structures (e.g. connecting them, using them as coordinates for more complex objects, intersecting these objects, …)
Finding the contour of an object
Now we combine Kinect`s depth information with contour detection of the openCV-Library (see the first part of this lecture for a brief introduction to openCV)
It`s even easier to use the library blobDetection by Julien Gachadoat.
import blobDetection.*;
import gab.opencv.*;
import org.openkinect.processing.*;
BlobDetection theBlobDetection;
Kinect2 kinect2;
PImage BWdepth;
void setup(){
size(640, 480);
kinect2 = new Kinect2(this);
kinect2.initVideo();
kinect2.initDepth();
kinect2.initIR();
kinect2.initRegistered();
kinect2.initDevice();
BWdepth = kinect2.getDepthImage();
theBlobDetection = new BlobDetection(BWdepth.width, BWdepth.height);
theBlobDetection.setPosDiscrimination(false);
theBlobDetection.setThreshold(0.1);
theBlobDetection.computeBlobs(BWdepth.pixels);
}
void draw(){
BWdepth = kinect2.getDepthImage();
theBlobDetection.computeBlobs(BWdepth.pixels);
image(BWdepth, 0, 0, width, height);
drawBlobsAndEdges();
}
void drawBlobsAndEdges() {
Blob b;
EdgeVertex eA, eB;
for (int n=0; n < theBlobDetection.getBlobNb(); n++){ // loop through all found blobs
b = theBlobDetection.getBlob(n);
float blobWidth = b.w; // get width and height of current blob
float blobHeight = b.h;
if (b!=null && blobWidth > 0.4 && blobHeight > 0.6){ // if it`s a huge blob draw its contours
strokeWeight(3);
stroke(0, 255, 0);
fill(255,0,0);
//PShape bs;
//bs = createShape();
//bs.beginShape();
//bs.fill(0, 0, 255);
for (int m=0; m < b.getEdgeNb(); m++){ // loop through all vertices of current blob
eA = b.getEdgeVertexA(m);
eB = b.getEdgeVertexB(m);
if (eA !=null && eB !=null)
//bs.vertex(eA.x*width, eA.y*height);
//bs.vertex(eB.x*width, eB.y*height);
line(eA.x*width, eA.y*height, eB.x*width, eB.y*height);
//bs.endShape();
//shape(bs, 0, 0);
}
}
}
}
How could the algorithm be impoved in a way to detect a body like in Marko Cicilianis piece "Steina"?
- Most important: apart from processing depth information, the setup of the observed scenery is crucial.
- prepare the images before detecting blobs, filter certain ranges of grey so blob-detection gets more robust
- use just the biggest blob
- cluster blobs and treat them as a single blob
To do:
- calculate the average distance of the contents of a contour to calculate distance of an object
- Controlling a MAX instrument with movement tracked by Kinect