Turns out is doing well enough to support decent aquatic life including the rare Olympia Oysters. In fact the Gorge Waterway has no closures I can find for shellfish, crab or fish. The Victoria harbour has a limit - from DFO:
Dioxin contamination - Victoria Harbour
Consumption of crab hepatopancreas harvested in those waters of Victoria Harbour north of a line from Macaulay Point to the navigation light at the western end of the Ogden Point breakwater to a line from Chapman Point southwesterly to the opposite shore should not exceed 135 g/week.
Fishing is something you can see a lot of off of the Craigflower bridge.
The Gorge may have been a problem in the past, but that is no longer the case. The Victoria Harbour still has issues, but the harbour area and the Gorge Waterway are not the same place.