The 28-year-old fashion designer is said to be battling the debilitating bacterial illness, which is caused by tick bites and causes pain in the sufferer's skin and joints and problems with the heart and nervous system, due to bites she received from the parasitic arachnids several years ago.
A source close to the actress told OK! magazine: "She's really sick. It's gotten worse. She was diagnosed in the very late stages so early detection measures weren't options for her. She's really going through it."
The illness has taken a physical toll on Ashley - the twin sister of Mary-Kate Olsen and older sibling of 'Avengers: Age of Ultron' star Elizabeth Olsen - in recent months and has affected her ability to work.
The source added: "When she does come to work, she looks haggard and disheveled. And she's often moody. Ashley's having a very rough time right now."
Pop singer Avril Lavinge has also been suffering from Lyme disease and recently admitted she thought she was "dying" when she was stricken down with the disease.
The 'Complicated' hitmaker said: "I felt like I couldn't breathe, I couldn't talk and I couldn't move. I thought I was dying ... I had no idea a bug bite could do this. I was bedridden for five months."