The libraries are virtual locations, they don't exist physically, they are organizational groupings. If you delete the files from the first location they will also be deleted from the second - the files aren't actually out there twice.
Libraries are a "feature" for your "convenience" (confusion).