Tamako's father died when she was seven years old, and she was raised by the retired Emperor Shirakawa and his favorite mistress Gion no Nyōgo (祇園女御) (Gion no Nyogo was Taira no Tadamori's wife also Mother of Taira no Kiyomori). Once she grew up, she was almost engaged to the heir of the regent Fujiwara clan, Fujiwara no Tadamichi, but his father Fujiwara no Tadazane incurred Shirakawa's displeasure by declining the offer due to rumors about Tamako's behavior.