Outdoor climbing may not be the only issue but it is by far the bigger issue. On flat ground the physics mean weight’s only an issue when accelerating up to speed, once you’re there the impact is marginal if any at all. The far bigger impact is wind resistance and a heavier rider with more power to push generally does better in the wind.
Once you’re at speed, think of an analogy to rowing. Weight absolutely has an impact on the speed of the boat, bigger impact than with cycling because your base coefficient of friction with the water is generally greater than the friction of your wheels, and I believe increases more with weight. Yet despite this on average, all else equal, crews with larger individuals are faster, because they are stronger.
Accelerating to speed will be an issue sure, so a course with lots of sharp corners that make you bleed speed could be an issue. But this can be mitigated with technique and note, successful crit racers in the US, CXers and MTBers are often / generally meaningfully larger than top GC racers, again because although they are big, they are also strong, they go faster on the flats and even the negatives / acceleration issues are not enough to outweigh this.
Don’t lose heart if you like longer endurance events, the best classics risers also tend to be on the bigger side because they are strong and tough and spend a lot of time in the wind. (Think Tom Boonen).
They key assumption here isn’t about what cycling events you want to do or duration, rather it’s the assumption that more size = more power. This is not always true but it very, very often is; more muscle equals more force production and greater anaerobic capacity.
So I’d suggest your path forward include the following:
I. Train for whatever events you enjoy the most
II. Don’t spend all your time bench pressing / skipping leg day
III. Most important, do not get the idea that just because you’re big, you can’t also be fast. Learn the situations in which your strength works for you and in which it’s a hindrance, and learn to maximize the former and mitigate the latter.