Recently I ran into following problem: e-mail alerts were not delivered to SharePoint users which were not belonging to the domain of the SharePoint server. The installation was consisting of a web application with windows authentication. This web application was extended with one with forms based authentication. Users logging in to the first web application (with Windows authentication) received perfectly their alerts. Users logging in to the second web application (with forms based authentication) didn't receive any alerts, nor did they receive the notification when they had set up a new alert.
Looking in the log files on the SharePoint server, I found following error:
#160009: The e-mail address 'firstname.lastname@example.org' is unknown.
The domain mentioned in the error was off course another domain than the domain of the users logging in through Windows authentication, as the second web application was created to allow access for external (it is, not belonging to the company) users.
We fixed this by adding the IP address of the Microsoft Office SharePoint Server 2007 into the Allowed Relays in Exchange.