I can't remember a single time [my parents] ever told me not...
I can't remember a single time [my parents] ever told me not to do something I wanted to do.
I can't remember a single time [my parents] ever told me not to do something I wanted to do.