https://mlart.co/?page=1

https://experiments.withgoogle.com/semi-conductor

Semi-Conductor

Semi-Conductor is an experiment that lets you conduct your own orchestra through your browser. You can move your arms to change the tempo, volume, and instrumentation of a piece of music. It uses PoseNet, a machine learning library that works in the browser, to map out your movements through your webcam. An algorithm plays along to the score as you conduct, using hundreds of tiny audio files from live recorded instruments.

In this experiment, the creators employed a body pose model. For the model’s training, they likely used a large number of images capturing conductors’ movements—specifically, how an orchestra responds to a conductor’s distinct gestures, such as playing louder or softer, or faster or slower. Arm movements play a pivotal role in orchestral conducting. Conductors primarily guide the orchestra through the coordinated work of both hands: the right hand typically holds a baton to mark the beat, rhythm and dynamic contours, while the left hand cues the entry of each instrumental section, controls the musical layers, and shapes the expression and dynamic shifts. Through the force, range and movement trajectory of their arms, conductors convey clear musical intentions and emotions, ensuring precise ensemble playing by the orchestra. Therefore, to capture arm movements and trigger corresponding responses from a digital orchestra, the creators opted to use a body pose model.

https://editor.p5js.org/JuliaQi/sketches/0TxP0IHDW

https://drive.google.com/file/d/1qWGrkrfpCNUeLx3MIhjgzI0yatM7MSi0/view?usp=drive_link

Inspired by Spider-Man, I created this sketch. When making a specific hand pose—folding middle and ring fingers—web strings will shoot out from wrists.

To detect this specific hand pose, I set a maximum threshold for the distance between the middle/ring fingers and the wrist, and a minimum threshold for the distance between the index/pinky fingers and the wrist.

    let hand = hands[i];
    let fingerTwo = hand.index_finger_tip;
    let fingerThree = hand.middle_finger_tip;
    let fingerFour = hand.ring_finger_tip;
    let wrist = hand.wrist;
    let d1 = dist(fingerThree.x, fingerThree.y, wrist.x, wrist.y);
    let d2 = dist(fingerFour.x, fingerFour.y, wrist.x, wrist.y);
    let d3 = dist(fingerTwo.x, fingerTwo.y, wrist.x, wrist.y);
    
    if (d1 < 100 && d2 < 100 && d3 > 100  && millis() - shootStartTime[i] > shootInterval) {
      triggerShoot[i] = true;
      shootStartTime[i] = millis();
      lineProgress[i] = 0;
      shootDir[i] = wrist.x < width/2 ? 'leftTop' : 'rightTop';
      shootOrigin[i].x = wrist.x;
      shootOrigin[i].y = wrist.y;

Using the center line of the screen as the boundary, when the hand is on the left side, web strings shoot toward the top-left corner; when the hand is on the right side, they shoot toward the top-right corner.

shootDir[i] = wrist.x < width/2 ? 'leftTop' : 'rightTop';

let targetX, targetY;
if (shootDir[i] === 'rightTop') {
  targetX = width;  
  targetY = 0;      
} else {
  targetX = 0;      
  targetY = 0;      
}

Complete Code: