Implementation / Software

We used an arduino uno to interface with the motor drivers, encoders, and ball dropping servo. The arduino communicated to a laptop through serial communication that controlled the entire system. In addition to the arduino, the laptop was also connected to a webcam which was used for detecting the target.

Sensing

The sensing algorithm was written using OpenCV. When the robot boots it takes a reference photo. During operation, the system computes the difference between new images from the reference photo. Each pixel in the reference photo is put through a hand tuned threshold to compute a binary mask displaying the new parts of the image in white. Using OpenCV’s contour detection algorithm we detect the shape with the largest area in the sensing region (outlined in green in last image) and consider it our target. We then use a linear mapping to determine the real world height of the object from the pixel location.

The arduino was used as a bridge between the laptop and hardware. We used hardware interrupts on the arduino to detect the size of the pulse modulated signals from the encoders. Using this technique after trying pulseIn we lowered the encoder’s error from 2 degrees to 0.75 and got a higher rate of communication as pulseIn, by design, waits for a whole wave from the encoder to be received. For the ball dropping mechanic we used the built in servo library to move the servo between 90 and 0 degrees to open and close the ball drop. To control the double pendulum the arduino sent 3 signals to each motor driver. The first two encoded the direction to apply torque to the motor. The third sent a varying 0-5 volts to the motor driver which mapped to 0-5 and 0-4 amps on the elbow and shoulder motors respectively.

The arduino communicated to the laptop over the serial usb cable. We tried several different protocols but arrived at using a continuous stream of information separated by a unique sequence of bytes that cannot occur in the data payload. Both the laptop and arduino wait for the unique sequence of bytes then read and decode the payload. The payload from arduino to server consists of:

5 header bytes (0000)

4 byte timestamp
1 byte padding (1)
2 byte shoulder angle (0-359)
2 byte current reading from elbow motor driver
1 byte padding (1)
2 byte elbow angle (0-359)
2 byte current reading from should motor driver
1 byte padding (1)

the payload from laptop to arduino consists of:

1 header byte (0)

1 byte shoulder direction (clockwise/counter clockwise/off)
1 byte padding (1 if shoulder current = 0, 2 otherwise)
1 byte shoulder current (1-255)
1 byte elbow direction (clockwise/counter clockwise/off)
1 byte padding (1 if elbow current = 0, 2 otherwise)
1 byte elbow current (1-255)
1 byte ball drop (open = 1, closed = 2)

Using this protocol we were are able to achieve a communication rate of ~300Hz and control the entire system from a laptop.

Interfacing

Executing Batting Plan

The batting plan consists of 5 parameters: drop time, shoulder angle, elbow angle, elbow direction, and should direction.

The batting plan is executed in a simple state machine. The state transitions through

SET_START_ANGLE
WAITING_FOR_DROP
WAITING_FOR_POS
BATTING
DONE

1. SET_START_ANGLE

In SET_START_ANGLE we use shoulder and elbow elbow angles from the batting plan to move to the initial configuration. We do this with the controller derived below.

The dynamics of double pendulum can be written as follows;

where,

To control our robot we tried to compensate for nonlinear terms with a feedforward controller s.t.

where,

is a nominal torque constant. We found that angular velocities sometimes showed large values due to noise, therefore, we neglect term in our trial.

Then, we tried to identify a plant model with the frequency response method. Since motors have a large damping factor due to the belts which connect motor and robot arm, we cannot assume the model which only includes inertias of the robot arm. According to the results, we modeled the plant as a 2nd order system.

Elbow:

Shoulder:

In the graphs, data points from experiments are shown in red circles, blue lines are estimated models as a 2nd order system. Based on these data, we set the PI controller such that

And tuned alpha and beta through simulation and experiments.

Once both the shoulder and elbow are within 5 degrees of the target angles we move to the next state WAITING_FOR_DROP.

2. WAITING_FOR_DROP & 3. WAITING_FOR_POS

In WAITING_FOR_DROP we wait for the user to send the fire command then move to the WAITING_FOR_POS state. In the WAITING_FOR_POS we apply the controller form SET_START_ANGLE and once both joints are within 5 degrees of the target we move to the BATTING state. The WAIT_FOR_DROP state can be arbitrarily long and joint angles may have changed by outside factors. By having this state we add more robustness to the system.

4. BATTING

During the batting phase we start a stopwatch and set the starting time to the batting plan’s drop time. We also set the ball drop the to open position to start the ball in a free fall. When the stopwatch time is positive, we apply maximum torque on both motors in the direction specified by the batting plan’s shoulder and elbow direction. When either the shoulder or elbow angles go past a specific threshold we move to the DONE state.

Below is the mathematical model that describes the batting phase given the batting plan. Simulating the mathematical at varying initial configuration gave us different parameters to try in our experiments which we added to the lookup table.

Inputting the maximum torque to the motors, we found two initial configurations, which enable the robot to hit a ball to the target at any height. At these configurations, we changed only the ball drop time in order to tune both contact angle and velocity of robot arm at the contact point. The simulation calculate the ball’s height given one of the configurations and the ball drop time. Procedures of the simulation is as follows:

Log experimental trajectory of robot arm according to time
Estimate contact timing, contact joint and the velocity at the contact point
Estimate height that the ball reach in the end

To develop the simulation, for the first step, we logged trajectories of the robot.

In the graphs, black points shows trajectory of hand, and red points shows trajectory of elbow.

Then, we estimate contact angle and contact velocity. We set a simulation time as follows:

Ball drops at time:
Robot arm actuates at time:
The ball and arm contacts at time:

is a control parameter that we need to define. is derived from the following calculations.

With these time stamps, we can predict the position of the ball at since it is free fall.

As we logged the position of robot arm according to time, we can also estimate the position of robot arm at

The contact timing will be the time at which the difference between and becomes the smallest, therefore,

where is a radius of tennis ball as a physical system cannot assume to be a point. With this time , we can find a contact angle and contact velocity.

In the last step, we estimated a height that the ball reach in the end. Based on basic dynamics, velocity of the ball after impact can be expressed as follows.

We set for coefficient of resitution of a tennis ball. With these velocities, we can estimate the heights.

In the graphs above, blue points show the position of the ball. The estimated height is 0.95[m] (Right) and 1.3[m] (Left).

5. DONE

In the done state we stop applying torque to both motors.

sensing

interfacing

executing batting plan