Heat energy flows from high temperatures to low temperatures by conduction, convection and radiation.
The wet cloth contains water(or any liquid),and by considering that liquids are heat conductors(poor heat conductors than solids) + while the dry cloth is trapping gas or air inside it, which are much poorer heat conductors than liquids.This will cause heat transmission much faster if the object is held using a wet cloth than a dryer one as liquids are better heat conductors than gases.
Hope I was useful.