arkemyr dont hit me

I went to the observatory, and did the whole business with the constellations before I went to talk to Arkemyr after I received the quest a cordial invitation. Now when I go to Arkemyr's manor he is always by his scrying pool and he just wants to duke it out when I see him. What should I doooooooooooo?