Statistics for The XXL survey XLII. Detection and characterization of the galaxy population of distant galaxy clusters in the XXL-N/VIDEO field: A tale of variety