There is confusion about Smith's birth year, with various sources citing either 1925 or 1928. Born James Oscar Smith in Norristown, Pennsylvania, at the age of six he joined his father doing a song-and-dance routine in clubs. He began teaching himself to play the piano.
When he was nine, Smith won a Philadelphia radio talent contest as a boogie-woogie pianist. After a stint in the navy, he began furthering his musical education in 1948, with a year at Royal Hamilton College of Music, then the Leo Ornstein School of Music in Philadelphia in 1949.
He began exploring the Hammond organ in 1951. From 1951 to 1954 he played piano, then organ in Philly R&B bands like Don Gardner and the Sonotones.
He switched to organ permanently in 1954 after hearing Wild Bill Davis.