When you add an image to Essential Graphics, I think it is also added as an item in your Project. If you can find the item in your Project window, you can replace it by right clicking on it and choosing Replace Footage.
Annoyingly, there doesn't seem to be a "Reveal In Project" option when you right click on an item in the Essential Graphics --> Edit pane, like there is when you click on a clip in a sequence.
If you are working with After Effects graphics templates, I'm not sure the individual items are added to your Project window, as I think those are designed to only be adjusted in After Effects.
Screenshot showing the "Replace Footage" option in Project window -> Select the image one wishes to change -> Right click on it: