why does he stop edward from killing james in the first book?

I think he stopped him from killing James was partly because of Bella, and if something happened to him & Bella were to pull through she would have been like she was in New Moon but worse. Also, he was so angry at James that he might have lost control and tortured him more than he should have been, and Carlisle is against hurting others unreasonably.
This also might cause much upset with the nomads, as the 'mate for a mate' quote goes. They would have delibratly set out to kill Bella. [ even though Laurent did in New Moon and Victoria did in Eclipse but you know what I mean aha. ]

Cause Carlisle is against killing any creature, and thinks very high of Edward. He doesnt want him to forget that they all decided to live in peace, and in a different way than the nomads. And the fact that Bella needed Edward was a factor aswell :D

Carlisle has taught him not to kill other creatures, he knows that Edward isn't a killer. To quote "This isn't who you are"

Cos he needs edward to suck out the venom from when james bites bella

he stops him because he knows that Edward is the one that needs to save bella