ike-cfg: Only consider port information if the IP matches

Otherwise, all configs would be considered to even some degree as the
ports usually match.

Closes strongswan/strongswan#2441

Fixes: 9228a5109b ("ike-cfg: Consider port information in IKE config match")

src/libcharon/config/ike_cfg.c

@@ -259,7 +259,7 @@ static u_int match(linked_list_t *hosts, linked_list_t *ranges, uint16_t port,
 	enumerator->destroy(enumerator);
 
 	/* honor if port matches exactly */
-	if (port == cand->get_port(cand))
+	if (quality && port == cand->get_port(cand))
 	{
 		quality += 1;
 	}