I can't be sure without seeing but is likely the paper backing. The vinyl peels off easy enough but the paper and glue stay behind. That's likely you're roughness left behind. I've donet this a few times. Absolutely nothing is an easy way, but the most luck I've had is with hot water and liquid fabric softener. Get a big spray bottle and fill it with the hottest water you can stand and then pour in some fabric softener and shake to mix thoroughly. While the water is hot spray the area you want to remove. When I say spray I mean soak it good. You'll want to put old towels or rags along the baseboard under it. Let it sit for about a minute (to soften the paper) and then take a 6" putty knife or scraper and scrape it off. It's tedious but it's the best way I've got it off including renting a steamer.
If the roughness is left behind by glue only, buy a couple of sanding blocks and sand it off. Be sure to wash it down real good if you plan to paint over it.