It is best to keep track of what control you've added to the form. This will make it easier to remove them back afterwards.
Here is an example that uses a stack (last in first out principle) to store a reference to the controls you've added.
By using the Pop method, you can retrieve the latest added control. You need to remove the control from the Controls collection of the form before disposing it.
NOTE: The sample is written in 2.0, let me know if you are using 1.1 so I'll rewrite it without using Generics.
private Stack<Control> _dynamicControls = new Stack<Control>();
private Point _lastLocation = new Point(50, 50);
private void btnAdd_Click(object sender, EventArgs e)
TextBox newTextBox = new TextBox();
newTextBox.Location = _lastLocation;
private void btnDelete_Click(object sender, EventArgs e)
if (_dynamicControls.Count > 0)
Control controlToBeRemoved = _dynamicControls.Pop();
Let me know if this isn't clear.