Yes, the email address must be present within the Contact Card (as Email 1, 2, 3 or Old Email Type) for the respective correspondence to/from that address to be linked to the record. Once the address is removed from the record, all correspondence from the address will unlink from that record and no longer be visible within Email History for that record.
We suggest to users to utilize Email 1 for primary email addresses, Email 2 and 3 for secondary email addresses, and Old Email for inactive addresses. In the new interface, you are not limited to just a total of 3 addresses as you can actually enter multiple occurrences of each type.
For detail information on how the archiving works, please click here.