In 1798, President John Adams appointed Sargent first Governor of the Mississippi Territory with its capital at Natchez. An impartial administrator, his contributions to the territory included authorship of its legal code; however, Sargent was not personally or politically popular. A contemporary declared: "However good his intentions, it is impossible that a man so frigid and sour can give satisfaction to a free people." Federalism was generally unpopular on the frontier. Thus, in 1801, newly elected Thomas Jefferson removed Sargent as governor.

Sargent chose to remain in Natchez where he became a cotton planter. His first wife died in 1790 in the first year of their marriage; he had remarried a wealthy Natchez widow on whose inherited estate they enlarged a house, rechristened (and, for the benefit of his territorial neighbors, re-spelled) "Gloster Place" after his hometown. In 1820, three years after Mississippi became a state, Sargent decided to move his family to Philadelphia. During the first leg of the journey, while on a steamboat to New Orleans, he died. His widow returned to Natchez, where Sargent is buried, but she apparently soon moved to Philadelphia.

Although Sargent's portrait and the pendant of his wife have usually been dated c. 1805, it is perhaps more likely that they were painted during the years of Sargent's governorship and after his marriage (October 24, 1798), c. 1799 - 1801. In that case, they would have been painted in Philadelphia where Stuart remained until 1803 when he went to Washington for two years.

The portrait commemorates the revolutionary officer and territorial soldier in his uniform of the Continental Army with the Order of the Cincinnati on his lapel. The sword, its blade held under his left arm, is more a symbol of authority than of war, perhaps suggesting Sargent's governorship. The strongly modelled head and the candid pale blue eyes would convince us of his intellect even if there were no other evidence which there is.

In some respects, as interesting as his professional career was, Sargent's avocation was as a scholar with diverse intellectual pursuits. A member of the American Philosophical Society and the American Academy of Arts and Sciences, he contributed papers both on natural and historic phenomena of the frontier territory that had become his home. The papers included a "List of Forest and Other Trees Northwest of the River Ohio" (Memoirs of the American Academy of Arts and Sciences, 1793), "An Account of Several Shocks of an Earthquake" (Memoirs, 1815), and, of particular interest, "Papers Relative to Certain American Antiquities" in which he reported his observations of an Indian mound near Cincinnati (delivered to the American Philosophical Society in Philadelphia in 1796, reprinted in their 1799 Transactions). It is clear that Sargent was equally a child of the Revolution and the Enlightenment, at home on the western frontier and in the eastern cosmopolis.