Do you have an example of using a custom edit form with the set_jq_editurl? You also mention in the documentation that you can use the 3rd parameter of the enable_edit but I don't see any examples
The quickest way for custom edit is to make a copy of edit.php (e.g. edit_custom.php) and add your own routine and logics after using set_jq_editurl(). The drawback is that the edit.php internal logic is not published (never intended to), so might take a few trial and errors.
Another approach, perhaps better, is to use an iframe for load a custom form. This is useful because the custom edit script could be on a remote site. The down side of this method is you cannot use PHP session due to cross-domain scripting.
On https://demo.phpcontrols.com, you can find 'Custom edit in iframe' demo that explains it in detail.