Have you considered switching the ISP1761 handler to request_threaded_irq() with IRQF_ONESHOT | IRQF_NO_SUSPEND so it runs in process context with that IRQ masked off, until completion?
That's something that Will suggested, but no - I didn't try it. This may be worth discussing with the ISP1761 driver maintainer...
Cheers!
Paweł