A difficulty in answering this question is dealing with there being so many possible conditions of isolation. The isolation of a child locked in a closet would be something very different from the isolation of a scientist who chose to work as a lone occupant of a space station. The isolation of a person who lives alone out in the country but spends most of the day interacting on the internet ... would that person really be isolated?

Since you are asking about "extreme isolation," I'll look primary at studies that involve sensory deprivation and/or loss of control over sensory input as experimental conditions...

RE: experiments studying the effects of isolation ... it's no surprise we might find some insight on this from the CIA Training Manual: "Solitary confinement acts on most persons as a powerful stressor. A person cut off from external stimuli turns his awareness inward and projects his unconsious outward. The symptoms most commonly produced by solitary confinement are superstition, intense love of any other living thing, perceiving inanimate objects as alive, hallucinations, and delusions. Deprivation of sensory stimuli induces stress and anxiety. The more complete the deprivation, the more rapidly and deeply the subject is affected. Some subjects progressively lose touch with reality, focus inwardly, and produce delusions, hallucinations, and other pathological effects." (This info quoted from an excellent APA paper on Isolation and Sensory Deprivation ... see the first of my sources below for full text).

The controversial experiments of Donald Hebb (former president of the American Psychological Association) in the '50s are often cited in articles on the effects of extreme isolation. The BBC recently revisited one of his sensory deprivation studies, and produced a documentary on the experiment. Six people were taken to a remote nuclear bunker and assigned to prison-quality rooms for 48 hours. Three of the people were left alone in the dark, windowless, and silent rooms; the other three were put in similar conditions but subjected to white noise and given foam armcuffs and goggles. (Video with highlights from the experiment can be found at the bbc link on my sources list). Test subjects reported hallucinations, and it was explained that this happens because the brain seeks to process information, and in the absence of significant sensory input the brain will generate it's own reality in order to create information to process. Test subjects from the first group were also shown to compulsively pace back and forth through their tiny rooms after less than a day of isolation. The subjects were given a series of psychological tests before and after their 48 hours in the bunker, and the tests revealed a deterioration of mental functioning after the experience of isolation.

Researcher Harry Harlow carried out a controversial and long series of experiments in the 1970's using monkeys, with a goal to understand the effects of isolation upon them. In his "pit of despair" experiments, he put monkeys into specially built chambers with steel walls. From a wiki article on this experiment (linked in my source list): "According to Harlow: 'most subjects typically assume a hunched position in a corner of the bottom of the apparatus. One might presume at this point that they find their situation to be hopeless.' Steven Suomi, one of Harlow's many doctoral students, placed some monkeys in the chamber for his PhD. He wrote that he could find no monkey who had any defense against it. Even the happiest monkeys came out damaged. He concluded that even a happy, normal childhood was no defense against depression. "

After reading various studies, the effects/experimental results I find coming up again and again are hallucinations, anxiety, repetitive physical behaviors, self-destructive behaviors, loss of mental acuity and extreme depression.