This can be found in the most sensational collection of songs, Prison And Mountain Songs (For Boys)
Most of the song titles listed on the back of Saint Dude are references to older, actual sheet music songs. This one, though, best I can tell, is not. It's just put in there since this is a collection of prison songs. I'm surprised though! A phrase like this seems like it would have a ton of songs about it, but I couldn't find any.