When a field is deleted in the plugin admin, it is only deleted from the fields database, not from the main database. If you want to delete the corresponding fields from the main database too, go ahead. Just make sure that for every field that is defined in the fields database, there is a matching column in the main database.
There are several columns you should not delete, of course: all the columns in the "internal" group.
Back up your database, and if things go wrong, just restore it.
