Theres a bunch of different methods. The one I used most frequently for Deadfire watercolours is this method:
1. Start with a portrait you want to outline like so:
2. Convert it to greyscale and duplicate the layer so you have a lower and upper layer that looks like this:
3. On the upper layer, invert colour and change the layer mode from "Normal" to "Dodge". Your layers should look like this:
4. ...and when summed together, you should get a white image like this:
5. Still on the upper layer, use a blur filter (gaussian blur, surface blur or smart blur). Now play around with blur radius and delta. Below are three different passes with different radius/delta values:
6. You can also combine different edge detect layers by adding, subtracting and dividing them into each other with layer modes. There are also some interesting ready made filter effects that produce interesting edge detects. GIMP for example has an artistic effect called "Photocopy" which produces an edge detect like this:
7. When I have some good edge layers, I open them in Inkscape and trace bitmap. I use multiple scan (grey), stack layers, no smoothing and remove background. You will get results that look like this:
8. When exporting your vector outlines from Inkscape, you need to save them as encapsulated postscript (.eps) or scalable vector graphics (.svg) files. This is because the vector outline can scale to any size but Photoshop wasn't made for editing scalable images (which is what Illustrator is for), so it converts vector images into raster images. These get pixelated when re-scaled. When you re-import your .eps/.svg files back into Photoshop, you will be asked what size you want the image to be.
Don't worry too much about the visual noise when doing various types of edge detect. You can remove this manually if you have to. I usually don't remove it with a black and white brush because this is very time consuming and often not necessary.
What I typically do is add my edge layers and create layer masks (full transparency) for each image. Then I "paint" into the layer mask everything I want to be opaque. Or if its less time consuming I will do it the other way around - set my layer mask to full opacity and paint into the layer mask everything I want to be transparent. This lets you choose what lines and what noise you want to be present in the final layer and it is completely non-destructive.
I find that the noise can help to achieve certain types of water colour or oil texture effects, so if it looks good, ill keep the noise and erase the outlines using a layer mask. Experiment with it.