Silverstein was born in Bedford-Stuyvesant, in Brooklyn in 1931 into a Jewish family. Growing up, Silverstein enjoyed classical music and played the piano. He attended the High School of Music and Art in New York, and then New York University, graduating in 1952. During college, Silverstein worked at a summer camp, where he met his wife, Klara. The couple married in 1956, and had three children: Lisa, Roger and Sharon. His wife worked as a school teacher, supporting the family on her salary for the first few years of their marriage while Silverstein attended classes at Brooklyn Law School.
Silverstein became involved in real estate, together with his late father, Harry G. Silverstein, and then friend and brother-in-law, the late Bernard Mendik, buying buildings in Manhattan. In 1957, they established Silverstein Properties, as Harry G. Silverstein & Sons, and bought their first building. Mendik and Silverstein continued the business after Harry's death in 1966. In 1977, Mendik divorced Annette Mendik Silverstein, with the business partnership also splitting up at that time. Mendik also cited disagreements over real estate strategies, with Mendik wanting to buy buildings while Silverstein wanted to build.