Suppose you wanted to hand-roll an oil capacitor out of paper, oil, and aluminum foil. The capacitance is C = εA/d; ε₀ ≈ 8.8541878 pF/m, so with 100-μm-thick paper soaked in oil with a relative permittivity of 5 (though both mineral oil and vegetable oil are closer to 2, you get 0.44 μF/m². Two typical aluminum-foil rolls of 10 m × 400 mm (see file aluminum-foil.md) with an equal amount of paper will form a 3.5 μF capacitor. Mineral oil breaks down around 10–15 MV/m so this thickness is good to about 1000–1500 V.
A more polarizable liquid like glycerin (relative permittivity 41.2–42.5) could improve capacitance and energy density by an order of magnitude, if the dielectric strength doesn’t suffer; in fact it’s reported to be 165 kV/cm = 16.5 MV/m at 55°, slightly higher, and higher still at lower temperatures. Propylene glycol might be another appealing alternative. Like most polar liquids, both of these are hygroscopic and would therefore need to be sealed thoroughly against water penetration if water is to be avoided. However, I’m not sure that, even if anhydrous, you wouldn’t suffer the same streamer-formation problem found in water-dielectric capacitors, where upon prolonged exposure to high voltage, streamers (high-conductivity paths maintained by current flow along them) form through the fluid. Electrolysis is known to be a problem with glycerin, presumably as a result of ionic contamination, but could possibly be suppressed with a thin insulating layer of glass.
To reduce water absorption, incorporating a stronger desiccant than glycerin into the capacitor was suggested by EEVblog user Zero999; coppercone2 listed desiccant alternatives, though I suspect some of the stronger ones in their list might be able to deprotonate the glycerin!
P2O5 >> BaO > Mg(ClO4)2, CaO, MgO, KOH (fused), conc H2SO4, CaSO4, Al2O3 > KOH (sticks), silica gel, Mg(ClO4)2·3 H2O > NaOH (fused), 95% H2SO4, CaBr2, CaCl2 (fused) > NaOH (sticks), Ba(ClO4)2, ZnCl2 (sticks), ZnBr2 > CaCl2 (technical) > CuSO4 > Na2SO4, K2CO3
Metal hydrides beat anything on the list, I think.
An important question for high-κ dielectrics is what their electrical relaxation time is; glycerol evidently is on the order of 10 ns, much longer than water but much shorter than the relaxation time of ion-movement relaxation mechanisms. This means that at higher frequencies a glycerin-dielectric capacitor would exhibit much lower capacitances. Small amounts of water in the glycerin speed this up enormously, down into the subnanosecond range with 20% water. For non-RF uses of capacitors this is adequately fast. Other researchers report much a slower relaxation time when encapsulated in silicone, in the 10 μs range, but they didn’t extend their dielectric spectroscopy to the sub-microsecond range, and it looks like the relative permittivity of their composite only dropped from about 20 to about 13.