How do you ensure that only intended recipient can decrypt file?

Here is the main role of contact list of every user. Which makes the decryption of a shared file possible only by the particular user you want. It means that the decryptions of the shared file from third party is impossible.