Optimising for navigation by making a view accessible shouldn’t come at the expense of the buttons in it not being accessible. Configuring custom actions will allow the user to access them using the Actions rotor.

With the Actions rotor, users can swipe up and down to cycle through all the options, stop at the desired one, and double tap to execute that action.
You can configure an array of custom actions. Custom actions can have a name that will be announced by VoiceOver and an action handler or target selector with the code that will be executed.
https://developer.apple.com/documentation/objectivec/nsobject-swift.class/accessibilitycustomactions
You may also find interesting...

Meet the rotor. A menu that you activate (and change options) by rotating two fingers on the screen. It lets you select different navigation modes and customizations. Like navigating through headings or changing VoiceOver’s speaking rate.

Touch target sizes are recommended to be at least 44 x 44 points. Buttons in the navigation bar ( especially when not using nav bar button items), dismiss buttons, and custom toolbars, are use cases that tend to have smaller sizes.
If you want to keep yourself up to date with what’s going on, or what has been published lately, on how to develop more accessible mobile apps, make sure you subscribe to Accessible Mobile Apps Weekly by @RobinKanatzar from @accessible_apps.