Peter Stuyvesant bought a farm, or bouwerie in Dutch, in 1651, which extended from what is now the East River to Lexington Ave., starting at 5th St. going up to 23rd St. The house itself, as pictured, was of yellow brick, located at Second Ave. and 10th St. The farm and orchard were worked by about 50 black slaves. Apparently, slavery was not an unusual institution in NYC at that time. The street known as "The Bowery" was originally a road leading to Stuyvesant's farm.
For some mind-bending information about the history of slavery in NYC, visit the Slavery in New York exhibit at the NY Historical Society, October 7, 2005 to March 5, 2006:
(Illustration from "The Wilderness to the Sea," by Benson J. Lossing. NY: Virtue and Yorston, 1866.)